Aditya,
   thanks for the information adding  \setmathfont[modern]   It fixed the problem for the default bullet symbols.  But, it did not fix the case where I'm trying to use the $\diamond$ symbol as the bullet, and compile time errors occur.  I will attach the log file.  Hopefully you can identify what is missing.

  I apologize for my lack of knowledge about fonts details and handling at this poing, but am learning.  Here is the minimal exampple:

--------------------------------
\usemodule[simplefonts]
\setmainfont[ipaexm]
\mainlanguage[ja]

\setmathfont[modern]

\definesymbol[5][$\diamond$]

\starttext
  \startitemize[5,columns,unpacked,two,broad]
       \item 此の行にbulletを打つ。
       \item One more column for good measure
       \item 此れは何処に置かれるでしょうか
   \stopitemize
\stoptext

----------------------------------------------------------------