The electromagnetic force compensation (EMFC) measurement principle has been widely adopted in the high-precision mass metrology system due to its sensitive compliant mechanism and nanometer level position sensor. In EMFC, an electromagnetic actuator balances the gravitational weight to maintain zero (or Null) position by feedback control using a position sensor, and the weight is calculated from the current applied to the actuator. Thus, a position sensor in the EMFC system should measure the null position accurately with high sensitivity and resolution. The position sensor commonly used in EMFC balance is an optical sensor that measures the displacement of EMFC balance from the intensity of light coming through a slit using a two-segment photodiode. This paper analyzed the characteristics of an optical position sensor for EMFC balance through parametric analysis using the Fresnel diffraction model. We also evaluated the performance of the sensor and confirmed the feasibility from weighing performance of the balance prototype. Normalized sensitivity of the sensor was 0.04237 μm^-1 and measured resolution was 1.09 nm. The weighing repeatability with our optical position sensor was 4.83 mg (1σ) at 10 g measurement, which was 3 times better than the repeatability with an alternative commercial sensor.