Lifetimes of chiral candidate structures in Rh103,104 were measured using the recoil distance Doppler-shift method. The Gammasphere detector array was used in conjunction with the Cologne plunger device. Excited states of Rh103,104 were populated by the 11B(96Zr,4n)103Rh and 11B(96Zr,3n)104Rh fusion-evaporation reactions in inverse kinematics. Three and five lifetimes of levels belonging to the proposed chiral doublet bands are measured in Rh103 and Rh104, respectively. The previously observed even-odd spin dependence of the B(M1)/B(E2) values is caused by the variation in the B(E2) values, whereas the B(M1) values decrease as a function of spin.