We report a polarization-multiplexed 320 Gbaud, 64 QAM coherent Nyquist pulse transmission with a frequency-stabilized mode-locked laser and a modified digital backpropagation method for pulse transmission. Using a combination consisting of a mode-locked laser and a pulse shaper, we obtained a Nyquist pulse with a high OSNR of 51 dB. We achieved error free operation under a back-To-back condition with the OSNR improvement. By developing a new digital back-propagation method for pulse propagation, we achieved a bit error rate below the 7% forward error correction limit of 2x10-3 for all the tributaries of the OTDM signal data after a 150 km transmission. As a result, single-channel 3.84 Tbit/s data were successfully transmitted over 150 km with a spectral efficiency of 10.6 bit/s/Hz.