Polarity-controlled ZnO thin films grown on different buffer layers were investigated as nonlinear optical materials for second harmonic generation. The effective nonlinear optical coefficient (deff) of ZnO grown on Cr-compound buffer layers showed a higher value than that of ZnO grown on MgO buffer layers. The correlations among the grain size and surface roughness with the values of deff were found to be strong. The deff of the ZnO film increased with decrease in the grain size and surface roughness. This relationship could be explained by the increase in reaction interface and the decrease in scattering probability at the surface as decrease in grain size and surface roughness, respectively.