Game theory is a branch of mathematics that uses models to study interactions with formalized incentive structures ("games"). It has applications in a variety of fields, including economics, evolutionary biology, political science, and military strategy. Game theorists study the predicted and actual behavior of individuals in games, as well as optimal strategies. From the Game theory article on Wikipedia.
(Also from wikipedia, the most commonly used beginners example in game theory: The prisoner's dilemma)
Here are a few other links on Game Theory and its applications:
Gametheory.net:a portal with many links to lecture notes,articles.
An interesting article, with examples of how game theory is used in FCC auctions, Intel/AMD price war and other applications.
And this is a link to a lecture [audio] from the Stanford GSB.