We have succeeded in growing single crystalline samples of CePdSi3 with the BaNiSn3-type non-centrosymmetric structure. Specific heat, magnetic susceptibility, and magnetization measurements were carried out, which revealed three successive magnetic transitions at TI = 4.85(5) K, TII = 2.78(5) K, and TIII = 2.30(5) K at zero field. Below TII, a weak ferromagnetic component was observed, indicating a canted antiferromagnetic ground state in CePdSi3. The observed large magnetic anisotropy in CePdSi3 was explained in terms of a simple crystalline-electric-field model. From the magnetization measurements, we identified five (three) magnetic states in the H-T phase diagrams for the H ||  () direction. Interestingly, only three magnetization steps were observed for H || , suggesting the existence of the in-plane magnetic anisotropy in CePdSi3. We discuss a possible nature of the complex magnetic properties of CePdSi3.