How to measure magnetic field induced magneto-optical (MO) activity of nonmagnetic elliptical plasmonic nanodisks which rest on a dielectric substrate remains a challenge since the substrate contribute most of the overall MO which varies with light polarization with respect to the orientation of the nanodisks. Here, we present a spectroscopic characterization. We find that only when light is incident from the nanostructures' side with polarization aligned with one of the two symmetry axes, one can subtract the MO contribution from the substrate by an amount equal to that of a bare one. By a detailed polarizing transmittance measurement, we determine the orientation of the two symmetry axes of the nanodisks. Light polarization is then aligned along the axes, enabling measurement of the intrinsic MO activity of gold nanodisks, which is the overall MO activity subtracted by that of a bare glass substrate. The narrow line widths of the plasmonic resonance features in the MO spectra imply a potential application in refractive index sensing.