Incoherency of high-frequency seismic waves leads to the idea to analyze seismogram envelopes disregarding phase information for the source study. We develop an inversion method for estimating high-frequency (above 1 Hz) energy radiation from the earthquake fault and site amplification factors from observed mean square S-wave seismogram envelopes. The inversion is executed by using the S-wave envelope Green function from the onset to coda for a point shear dislocation source in a scattering medium, which is formulated based on an extended version of the radiative transfer theory. The use of the envelope Green function enables us to estimate the spatial distribution of high-frequency energy radiation on a fault plane. This has been difficult by conventional waveform inversion methods. The rupture velocity and the duration time of energy radiation on each subfault are also estimated by grid search method. We apply this new method to the 1994 far east off Sanriku, northeastern Japan, earthquake (MW 7.7), which ruptured a 90-km segment of the plate boundary along the Japan Trench. Inverting observed seismogram envelopes at 10 strong-motion stations in Japan, it is concluded that above 90% of the high-frequency energy was radiated from the western half of the fault with the largest energy radiation near the deep-side edge. The rupture velocity is estimated to be 2.7 km/s. The estimated site amplification factors, ranging between 0.3 and 15, are consistent with those independently estimated by the coda normalization method.