This paper describes a method to evaluate daily physical activity by means of a portable device that determines the type of physical activity based on accelerometers and a barometer. Energy consumption of a given type of physical activity was calculated according to relative metabolic ratio (RMR) of each physical activity type that reflects exercise intensity of activities. Special attention was paid to classification algorithms for activity typing that identify detailed ambulatory movements considering vertical movements, such as stair/slope climbing or use of elevators. A portable measurement device with accelerometers and a barometer, and a Kalman filter was designed to detect the features of vertical movements. Furthermore, walking speed was calculated by an equation which estimates the walking speed as a function of signal energy of vertical body acceleration during walking. To confirm the usefulness of the method, preliminary experiments were performed with healthy young and elderly subjects. The portable device was attached to the waist. A standard accelerometer based calorie counter was also attached for comparison. Experimental results showed that the proposed method feasibly classified the type of ambulatory physical activities; level walking, stair going up and down and elevator use. It was suggested that the consideration of vertical movements made a significant improvement in the estimation of energy consumptions, and the proposed method provides better estimation of physical activity compared to the conventional calorie counter.