Agile software development is a methodology for delivering software in a predictable, workable and meaningful way.
The gist is you favor people over process, collaboration over conflict. You write out your requirements in technology-neutral language, iterate quickly, release often, and get immediate feedback. That's it. Read the Manifesto for Agile Software Development and then figure out what that means for your team.
You need to do three things, and three things only. One, define what you're trying to achieve in a technology-neutral way, two, track how long it takes to get these things done so you can reasonably estimate future work, and three, reduce friction and drag wherever possible to keep a continuous flow of software delivery going, every week to two weeks. That. Literally. It.
The reasons for doing so are two-fold: the first is that software developers have an average career span of less than a year. Within one quarter of the time pursuing a software engineering degree, many folks bail from using it. People can only take so much nonsense, and keeping people around is key.
But the second is that the old-school "big design up front" just simply didn't work. It works in other spheres: if you're designing a building, you're working in a sphere where things take time and people have specific skills. The building is as the building is - you're designing, for example, a hospital - and you're not going to decide halfway through what you really needed was a parking garage. Likewise with the specialists involved in the construction - you'd never ask a plasterer to do electrical work, and you wouldn't ask a structural engineer to figure out your paint schemes.
One of the running gags in Life in Hell by the Simpsons' Matt Goering is that the gay lover pair Akbar and Jeff's place of business is doing something different every week. One week it's "Lucky Psychic hut" and the next it was "Lucky T-shirt hut". I know of at least two businesses run by millennial friends of mine - one of whose business was social networking marketing analysis one week, a restaurant finder app the next. No other type of business would tolerate changes like this, but then again, most enterprises don't go out of businesses as tastes change. Ask MySpace.
And in the latest zeal to get something for nothing, managers are now asking for "full stack" developers, in other words instead of hiring a backend developer, a front end developer, a UX designer, a graphic designer, a database specialist and a DevOps guy, they try to find some young prima donna wannabe who'll just live at work in a beanbag chair and do ALL of it for the price of one. But that is in no way new - the whole industry was started by hobbyists and many a product was created soup-to-nuts by one or at least a small team of people.
But that kind of change means that all manner of prior methods of designing a manufacturing process were just unworkable. Plainly unworkable. It was pretty much a given back in the day that about 80% of software projects would fail. By the time you'd had all the big meetings and more meetings and meetings meetings meetings, had your "systems architects" draft up the 30,000' view of the system unconstrained by actual problems on the ground - and your software developers finally got to work on the problem, assuming you got every prediction 100% right (which is practically impossible) - not only were your requirements obsolete and had to be redone, but there was a whole ripple effect. We needed some changes on the front end (which you worked on last) which meant changes to the back end (which was signed off on six months ago) and that of course means changes to the database and systems capacities (signed off a year ago). The guys in the wire rimmed glasses and metrosexual hair who wrote the "architecture documents" had of course happily fucked off to their next high-paying, no actual work gig. ("We need some clever code called ThisClassOfAwesomenessFactory right here"). So basically there was a back-ripple effect of complete fuckery. Everyone shrugged their shoulders, accepted that the past year had been a total waste of money, and repeated the same errors all over again.
Not to say that software didn't get written, and things didn't happen, but the cartoon Dilbert exists for a reason, and you had people figuring things out despite the system. Software requirements guys wrote in 200% of the actual requirements knowing only half of what was promised could potentially get done, and the engineers found ways to half-ass completion of features, or simply say "you didn't ask for that". It was a kind of oppositional combat of managers vs developers, with the bean counters stuck in the middle BEGGING to please find some way to eliminate all the waste in the system.
Enter Agile. To heck with all this, they said, look guys, let's take an approach where we design features the way people really do when they don't think of it as Real Serious Business. If you go to the Third World, for example, and people get tired of wading across a small river, someone will eventually put a plank across the river. That plank will be moved a couple of times but eventually someone will find a spot organically that people will be happy with. People walk across the plank quite happily, maybe replacing it when it breaks. But then someone will notice that they'd really like to get their cart over it, so they take some time and put four planks across, nail them together, and secure them to both sides, making a wooden bridge. After a while other cart guys use it too, so they might make two lanes, brace it a bit. Then people realize they'd like to drive a car across it, so they build a concrete bridge right next to the wooden bridge and deprecate the wooden bridge. As opposed to building a concrete bridge that never gets used because it's not in the right place, and who'd pay $5 to cross a river when walking across is easier?
And to be fair, some landscapers do the same thing these days. Tired of seeing their laid out paths unused and their grass trampled into paths where people actually chose to walk, a school of thought arose that said simply: plant some grass, after a year see where the grass is worn down, and build the paths there. You can always tell who doesn't do this, by simple visual inspection.
Agile said okay, let's solve a bunch of problems this way. Start with an initiative to do X. Build the very simplest thing that does the first step in a more complete solution to X. Track how long it took. Involve your users right from the beginning, show them nothing more than the site being up at first, but as time goes by, have them use what you're building as you build it. You know when to stop working on the feature when your users love the way it works, which can free you up from putting chrome on a lily to figuring out what your users really wanted - and often times they don't know themselves until they work with something. Many a time someone will guess that a feature is going to be a killer selling point, when in fact it turns out it's a pain in the ass and gets in the way of the worker being able to do what they need with it.
That's in essence what all the nice language and scrums and blah blah blah is saying. It's saying rather than guess what people want, involve them really early and build what they need. No more, no less. Track what you do so that you can estimate after a while how long it takes to do things.
And you would be AMAZED at how quickly people want to screw it up. To explain the concept of what Agile tries to achieve, I'll point out how people mess with it, and why that's bad.
Managers don't like it because when you take process-heavy work out of the equation, they're obsolete. Many of them have had whatever skill they had prior to going into management atrophy - their ability to code or even to design a working implementation obsoleted by the rapidly moving pace of technology while they were organizing meetings and drawing up Gantt charts in Microsoft Project. So they will argue there's no structure. There's no oversight. There's no big reports. There's no oversight. God forbid the users talk directly to the developers and between them work out what is needed. How can you possibly do that without 8 hours' worth of meetings and reports per day?
Architects hate it, because Agile presumes architecture will evolve, not be dictated. That means that the days of drawing pretty UML diagrams with class names and method names all happily laid out - gone. The days of a six figure check for writing a document nobody needs and is pretty much useless the moment all the graphs are drawn - done. You code or you find another way to make a living. And those guys went into picture drawing and "big picture thinking" because they didn't want to do actual work, and being asked to actually write code makes many of them shit themselves.
So naturally, they try "Scrumbutt". "Well, we do Scrum, but..." And the signs of this disease are that they do indeed have a Big Design Up Front, and that the "stories" in their backlog are not steps in a solution but "do paragraph A of my big design document", "do paragraph B of my big design document". The most egregious example I've seen was where the story asked for wasn't "As a user, I want to be able to enter this data so that I can perform an action" but "use this specific software component I have chosen because it looks pretty". The developer guessed at how the manager wanted it used, and guessed wrong. Even after the manager rolled his eyes and said it was a bug and thanks a lot for screwing up the schedule, and got it to the manager's specification - it wasn't what the users wanted and though it looked VERY sexy in a boardroom meeting, it was hated by the user base.
Another set of warning signs: they don't do Scrum meetings, which are simple heartbeats in the morning ("Still working on story 9. Nothing's blocking me" "Working on story 4, I'll need to talk to Mike to get this resolved, Mike, can we meet after?") and turn them into "status meetings". They don't meet every two weeks to see what went right and what went wrong, and improve their process ("talking to the client let us work out how to do this faster", "we didn't find that three people working per team was as efficient as four people per team") but instead do "root cause analysis" meetings. "Okay, here's how Mitch fucked up the multi-select, let's go do a code reivew and tell him everything he did wrong.")
There's actually a joke in Agile culture about "Agile baseball". The manager says "well, we tried playing baseball, but only one person gets to bat? Half the team's just standing there? Huh. That didn't really work for our team. Lot of people standing around doing nothing. So what we decided instead was rather than set it up as a DIAMOND, with bases, we'd just have everyone stand on a line and everyone gets to bat at the same time. Of course - um, this meant there was nobody to throw the ball, so we just had guys throw their own ball in the air and then hit it themselves. Of course, it meant that they'd have to go out and get their ball back, and anyway with all those balls being hit, running the bases seemed like a stupid idea given you had to go get your own ball back - took too much time to run the bases, go out and get your ball, come back. And Sandra was knocked out by Paul's line drive. Anyway, we couldn't figure out how to score it, it's just stupid and unworkable, so we went back to playing soccer."
But then again, company process is often like scar tissue: it builds up over problems and preserves people's jobs. Many a useless employee is perfectly happy to spend years and years at Big Technology Corporation happily filling in reports and taking his coffee mug to meetings, and nobody ever seems to notice that his time is spent doing things that relate in no way to the company making money.