In this work, a skutterudite-based compound, Yb0.25Co4Sb12, added with Alx (x = 0, 0.1, 0.2, 0.3) was synthesized with a simple mechanical alloying technique followed by spark plasma sintering. The microstructural properties and thermoelectric properties of the as-sintered samples were investigated. The Al atoms formed AlSb nano-inclusions in the grain boundaries instead of entering the Sb-icosahedral voids, introducing point defects in the matrix lattice. By scattering low-energy electrons, the grain boundaries acted as a potential barrier in simultaneously attaining low electrical resistivity and high Seebeck coefficient. Therefore, Al0.1Yb0.25Co4Sb12 exhibited a high power factor of 4.8 × 10−3 W/m K2 at 377 °C. AlSb of nanometer length enhanced interfacial phonon scattering, thereby significantly reducing the lattice thermal conductivity of Al0.3Yb0.25Co4Sb12 to 0.6 W/m K at 500 K. The Al0.3Yb0.25Co4Sb12 composite exhibited the highest figure of merit, ZT = 1.36, at 850 K.