Background. The sensory drive hypothesis predicts that divergent sensory adaptation in different habitats may lead to premating isolation upon secondary contact of populations. Speciation by sensory drive has traditionally been treated as a special case of speciation as a byproduct of adaptation to divergent environments in geographically isolated populations. However, if habitats are heterogeneous, local adaptation in the sensory systems may cause the emergence of reproductively isolated species from a single unstructured population. In polychromatic fishes, visual sensitivity might become adapted to local ambient light regimes and the sensitivity might influence female preferences for male nuptial color. In this paper, we investigate the possibility of speciation by sensory drive as a byproduct of divergent visual adaptation within a single initially unstructured population. We use models based on explicit genetic mechanisms for color vision and nuptial coloration. Results. We show that in simulations in which the adaptive evolution of visual pigments and color perception are explicitly modeled, sensory drive can promote speciation along a short selection gradient within a continuous habitat and population. We assumed that color perception evolves to adapt to the modal light environment that individuals experience and that females prefer to mate with males whose nuptial color they are most sensitive to. In our simulations color perception depends on the absorption spectra of an individual's visual pigments. Speciation occurred most frequently when the steepness of the environmental light gradient was intermediate and dispersal distance of offspring was relatively small. In addition, our results predict that mutations that cause large shifts in the wavelength of peak absorption promote speciation, whereas we did not observe speciation when peak absorption evolved by stepwise mutations with small effect. Conclusion. The results suggest that speciation can occur where environmental gradients create divergent selection on sensory modalities that are used in mate choice. Evidence for such gradients exists from several animal groups, and from freshwater and marine fishes in particular. The probability of speciation in a continuous population under such conditions may then critically depend on the genetic architecture of perceptual adaptation and female mate choice.