African tigerfish (Hydrocynus sp.) with large, conical teeth. Credit: Lucie Helešicová, Unsplash (CC0, creativecommons.org/publicdomain/zero/1.0/)

A trade-off between tooth size and jaw mobility has restricted fish evolution, Nick Peoples at the University of California Davis, US, and colleagues report in the open-access journal .

Ray-finned fish are a diverse and widespread group, representing 99% of living fish species. Two key adaptations have helped them thrive: large teeth and extendable jaws.

Fish with larger teeth can access a wider range of food sources, while the ability to rapidly extend the allows fish to hunt fast-swimming prey by using suction forces to pull them closer. However, these two innovations are rarely found in the same . To find out why, researchers captured high-speed videos of the feeding behavior of 161 species of ray-finned fish.

They analyzed how different strategies for capturing prey had evolved alongside tooth size across the evolutionary tree. They found that small-toothed fishes used a wider variety of methods to capture prey, including extending their jaw to create suction.

In contrast, fishes with larger teeth were more likely to use rapid bursts of swimming to close in on their prey, and fish with the largest teeth exclusively used this strategy. The researchers calculated that the optimum tooth size for fish that use jaw extension to capture prey was four times smaller than for fish that use rapid swimming.

A killifish (Aplocheilus lineatus) feeding on brine shrimp in the lab. Credit: Nick Peoples (CC-BY 4.0, creativecommons.org/licenses/by/4.0/)

The results indicate that large teeth and a highly extendable upper jaw are incompatible adaptations, creating a trade-off in the evolution of fish feeding behavior. Most ray-finned fish grow replacement teeth inside the jawbone, so tooth size could be limited by jaw size. But slender, mobile jaws are better suited to rapid extension.

This structural limitation may explain why fish that capture their with jaw extension are restricted to relatively small teeth, the authors say.

"When we looked into which species had exceptionally large ," Peoples adds, "we were surprised to find that they also have interesting modifications to the dental system, like the beak of parrotfish."

Peoples says, "Many feeding innovations have evolved in fishes, and our results show that it is important to consider the interactions between these traits as some may actually be incompatible with each other."

Boulengerella maculata, a characiform from South America, preying on a mosquitofish in the lab. Credit: Nick Peoples (CC-BY 4.0, creativecommons.org/licenses/by/4.0/)

"High-speed videography really provides a unique view into fish feeding," Peoples remarks. "It often reveals subtle movements and behaviors that you would not notice if only observing the fish in the aquarium."

Peoples notes, "Working with live fish in the lab is always one of the highlights of my week, but it's often challenging to get species feeding as naturally as possible."

More information: Nick Peoples et al, Incompatibility between two major innovations shaped the diversification of fish feeding mechanisms. PLOS Biology (2025).

Journal information: PLoS Biology