We develop the model for the terahertz (THz) and infrared (IR) photoconductivity of graphene layers (GLs) at room temperature. The model accounts for the linear GL energy spectrum and the features of the energy relaxation and generation-recombination mechanisms inherent at room temperature, namely, the optical phonon absorption and emission and the Auger interband processes. Using the developed model, we calculate the spectral dependences of the THz and IR photoconductivity of the GLs. We show that the GL photoconductivity can change sign depending on the photon frequency, the GL doping and the dominant mechanism of the carrier momentum relaxation. We also evaluate the responsivity of the THz and IR photodetectors using the GL photoconductivity. The obtained results along with the relevant experimental data might reveal the microscopic processes in GLs, and the developed model could be used for the optimization of the GL-based photodetectors.