The nonlinear optical properties of Rh-Pd and Rh-Pt solid-solution alloy nanoparticles (NPs) were experimentally investigated by means of the z-scan technique. The open aperture (OA) measurement showed a reverse saturable behavior, whereas the closed aperture (CA) measurement showed a peak- and valley-shape. Both the Rh-Pd and Rh-Pt NPs exhibited a positive nonlinear optical index of refraction at 800 nm relating to the self-focusing phenomenon. In addition, the nonlinear absorption of the Rh-Pt NPs (4.39 × 10−12 cm/W) was higher than that of the Rh-Pd NPs (1.63 × 10−12 cm/W) due to the small interval between the occupied and unoccupied density of states (DOS) of Rh-Pt than Rh-Pd. The nonlinear responses of the Rh-Pd and Rh-Pt NPs was attributed to the hot electron contribution and the reverse saturation of intraband and interband transitions.