The eddy current magnetic signature (EC-MS) method is a relatively new method used to evaluate the residual stresses of ferromagnetic steels with higher sensitivity than other magnetic techniques. In this study, the effects of mechanical polishing and the ferrite–pearlite structure on the shape of the EC-MS signal were investigated in two experiments to clarify the effect of compressive residual stress on the EC-MS signal shape. In the first experiment, the work-hardened layers of the specimens were removed by electrolytic polishing and the EC-MS signals of the specimens were investigated. The second experiment involved the EC-MS measurements of materials with different pearlite contents, i.e., low-carbon steel with 0.17% and 0.27% carbon and pure iron. Specimens with different residual strains were prepared using a tensile test apparatus and their EC-MS signals were compared. In both experiments, the residual stresses acting on the specimens were measured adopting X-ray diffraction. The results reveal that the non-anisotropic compressive residual stress acting on the specimen surface changed the EC-MS signal in an anti-clockwise manner while the unidirectional compressive residual stress inside the specimen changed the EC-MS signal in a clockwise manner.