I think there is an issue with the size of math fonts in \frac when used in math alignments, as it is is shown in the following example.

I confirm what you said.

However, if I use \dm{} instead of \displaystyle as following, then the output is correct.

\placeformula
\startformula
\startalignedcases
\NC - \Delta u \NC = 0 \NC \qquad\text{in }\, \Omega \NR
\NC  \dm{\frac{\partial u}{\partial {\bi n}}}
 \NC = V u + \phi \NC \qquad\text{on }\, \partial\Omega. \NR
\stopalignedcases
\stopformula
or this (with\type{\displaystyle} added)
\placeformula 
\startformula
\startalignedcases
\NC - \Delta u \NC = 0 \NC \qquad\text{in }\, \Omega \NR
\NC  \dm{\frac{\partial u}{\partial {\bi n}}}
 \NC = V u + \phi \NC \qquad\text{on }\, \partial\Omega. \NR
\stopalignedcases
\stopformula

In this example, it is OK to use \dm inside of \startformula … \stopformula.