A webhook is a callback in an API.
Okay, let's back up. Suppose I want to track my XP over time, I could write a program that downloads my home node each day, parses the HTML, and extracts the data I need. This is a bit complicated and inefficient — all that cruft making the beautiful human-readable webpage is just in my way. An API is an interface specifically for use by programs. If I could just download api.everything2.com/user/General+Wesc an get my user info in a computer-readable format, no extra cruft, it'd be much simpler. That's an API. It's an Interface between my Program, the Application that is E2.
It's still kind of annoying that I have to have my program download the data each day looking for changes. Sure would be nice if my XP tracker could tell the E2 API, "if General Wesc's XP changes, poke me". That's a webhook. It's a push API because instead of my program periodically pulling data from the API, the API pushes the data to me when it's appropriate.
This is called a webhook because in programming, a hook was a place you could pop into existing code and say "hey, do this instead/also". We're telling the API that whenever my XP changes, also send me the new value. (Wikipedia informs me that the term "webhook" was coined in 2007 by Jeff Lindsay.)
Webhooks are often left out when making an API. They're more a convenience than an essential. Getting data is the first thing you'd implement, and being able to modify data would be the second. Implementing the ability for my API to pingback anyone who wants to know when XP changes would mean adding a new type of object into my database ("who do I ping for what"), having a way to detect any change to XP (there are lots of places in the program that modifies XP), and having a thing that then runs any hooks registered for that behavior. Oh, and handling errors.
If a thousand people have said "Let me know whenever General Wesc's XP changes", that's a thousand requests it has to change every time my XP changes. You'll want to make sure that runs asynchronously so the rest of your code isn't blocked waiting. However, this still may be better than each of those people having a program downloading my API information every second of every day, so if you have API users who really want up-to-date information, then implementing webhooks may ultimately reduce the server load.