An area of behavioral ecology, the study of how organisms interact with their environment in learned and instinctive ways.
Optimal foraging theory draws on mathematical economics, modeling organisms (in particular predators) as if they were human consumers shopping for food. The basic idea is that prey species differ in energy content, nutritional value, abundance, ease of finding and catching, and so on. Foraging for prey might also involve risks, depending on how dangerous the prey items are to hunt and catch, and whether an organism is more likely to be preyed upon while foraging. As a result (ok, this is really the main idea) natural selection should have shaped predators' instinctive behaviors to an optimal diet.
Research in the area of optimal foraging theory generally involves what boils down to a cost-benefit analysis for the forager being studied, where costs and benefits depend on the organism's morphology. For instance, different prey species might cost a snake differently in terms of venom expenditure. On the other hand, larger prey might offer a greater reward in terms of energy obtained by their capture. The bottom line is evolutionary fitness, and one of the most theoretically difficult problems in optimal foraging theory is understanding how different factors like meeting or exceeding energy needs, minimizing risk of injury while foraging, and minimizing risk of predation all add up to fitness.
Some of the models developed by optimal foraging theorists include:
state-dependent theory, which takes into account the current state of the forager in analyzing its behavior. For instance, a hungry predator might take more risks in looking for food than one who has not eaten recently. If the word "state" made you think of state machines, it's ok: in fact the math relating to this area features lots of Markov Chains, which can be modeled in terms of stochastic state machines.
prey choice models, which focus primarily on how predators decide what to eat (or, perhaps, evolved a preferred diet). In studying the prey preferences of organisms, one generally observes a continuum of predator response to prey variance in size or nutritional value, rather than an all-or-nothing dichotomy. Yet many species exhibit switching behavior in response to prey frequency: that is to say they eat whatever food is most prevalent, and can detect with surprising accuracy what that food is.
patch choice models, which examine the ways in which foragers deal with spatial heterogeneity in resource abundance. At some point, more energy is spent on searching for food than is obtained by this search: at this point it would be advantageous to find a new patch to hunt or graze (or whatever) in. I spent a few hours last week watching bees move from flower to flower searching for nectar. For the life of me, I couldn't detect any patterns in their activity. But that's because a lot of factors affect patch selection behaviors. For example, if patches are few and far between, it might be best to leave a patch before it becomes too energy-expensive to forage there any more, when one still has energy reserves with which to find another patch. On the other hand, if predation risk is high, it's probably best to stay in one patch until it becomes more necessary to move. Charnov's Marginal Value Theorem states that the best time to leave a patch is when the marginal rate of energy return along the patch curve (i.e. the derivative of expected energy return) equals the long-term energy intake in that patch/habitat. (I said this borrows a lot from economics at the beginning!)
Sources: my ecology notes. There's probably further reading in The Economy of Nature by Robert E. Ricklefs, but I'm too lazy to crack open a textbook most of the time.