Caveolae are bulb-shaped nanodomains of the plasma membrane that are enriched in cholesterol and sphingolipids. They have many physiological functions, including endocytic transport, mechanosensing, and regulation of membrane and lipid transport. Caveola formation relies on integral membrane proteins termed caveolins (Cavs) and the cavin family of peripheral proteins. Both protein families bind anionic phospholipids, but the precise roles of these lipids are unknown. Here, we studied the effects of phosphatidylserine (PtdSer), phosphatidylinositol 4-phosphate (PtdIns4P), and phosphatidylinositol 4,5-bisphosphate (PtdIns(4,5)P2) on caveolar formation and dynamics. Using live-cell, single-particle tracking of GFP-labeled Cav1 and ultrastructural analyses, we compared the effect of PtdSer disruption or phosphoinositide depletion with caveola disassembly caused by cavin1 loss. We found that PtdSer plays a crucial role in both caveola formation and stability. Sequestration or depletion of PtdSer decreased the number of detectable Cav1-GFP puncta and the number of caveolae visualized by electron microscopy. Under PtdSer-limiting conditions, the co-localization of Cav1 and cavin1 was diminished, and cavin1 degradation was increased. Using rapamycin-recruitable phosphatases, we also found that the acute depletion of PtdIns4P and PtdIns (4,5)P2 has minimal impact on caveola assembly but results in decreased lateral confinement. Finally, we show in a model of phospholipid scrambling, a feature of apoptotic cells, that caveola stability is acutely affected by the scrambling. We conclude that the predominant plasmalemmal anionic lipid PtdSer is essential for proper Cav clustering, caveola formation, and caveola dynamics and that membrane scrambling can perturb caveolar stability.