Ampullary carcinomas are highly malignant neoplasms that can have either intestinal or pancreatobiliary differentiation. To characterize somatic alterations in ampullary carcinomas, we performed whole-exome sequencing and DNA copy-number analysis on 60 ampullary carcinomas resected from clinically well-characterized Japanese and American patients. We next selected 92 genes and performed targeted sequencing to validate significantly mutated genes in an additional 112 cancers. The prevalence of driver gene mutations in carcinomas with the intestinal phenotype is different from those with the pancreatobiliary phenotype. We identified a characteristic significantly mutated driver gene (ELF3) as well as previously known driver genes (TP53, KRAS, APC, and others). Functional studies demonstrated that ELF3 silencing in normal human epithelial cells enhances their motility and invasion. Yachida et al. conduct an in-depth genomic analysis on ampullary carcinomas from Japanese and American patient cohorts. This leads to the identification of ELF3 as a driver tumor suppressor and other potentially therapeutically targetable mutations.