For role-players, there are those of us who played Dungeons & Dragons in the past and still play and those of us who played Dungeons & Dragons in the past and have moved on to something else. The common denominator is that just about everyone who has played the game has a strong opinion. Most who have left it feel pretty good about their new game, possibly militantly so. So why is D&D so popular?
Market share. It's the same thing that maddens Apple people who can't get software for their Macintosh. If you want to play a role-playing game, you need players. If those players are your friends, you'll probably play what they play. If you're looking at the general public as your player base, D&D has about 60% market share. This means 6 out of 10 role players are up for a game of D&D, while maybe 1 in 10 will play the next most popular. Customers place notices on our game board for months at a time, looking for players for their non D&D games. It's simply harder to find players, and when those players leave, much harder to replace them. It's actually getting more strained as we have the role-playing "long tail" and the emergence of indie games. Want to get a Shadowrun game together? Version 3 or 4? They're equally popular. Want to play an indie game? You better have board game style role-players willing to learn a new game every week. You can't just say it's popular because it's popular. Why does D&D have this market share that drives players into its arms?
Fantasy as theme. I've yet to see a long term role-playing game survive based on anything other than fantasy. D&D does fantasy pretty well, it's what it was designed to do (it does everything else pretty poorly). I've played spy games, superhero games, science fiction games, Western games, and pulp games, and they have no staying power for the groups I've played in. Wow, that was fun, when is our next D&D session? At the same time, I see D&D groups last for years, decades even.
As far as imagination goes, there seems to be too much suspension of disbelief to play more modern games. Perhaps we're not removed enough from reality to get into the spirit? I read an article about European toy makers and how none of the modern toy figures have guns, but all the medieval figures have swords and armor. Police men go unarmed, but knights have a bewildering array of weapons. That can't be just about non-violence. Those ancient battles are resolved and their horrors are too far removed from our reality, which makes them fun and acceptable as instruments of play. A knight with a sword is fun and exciting, a Rwandan with a machete is horrifying and guilt ridden. Fantasy conflict is far removed from day to day horrors. A gun, be it a Colt Peacemaker, a laser pistol, or an Uzi, all bring up modern day horrors. Who do you know who was robbed at glaive point?
Finally, it's about mechanics. The same thing that draws people to D&D and makes them stick to it is the same thing that pushes others away. Character creation in D&D is a game in itself. Even before the "synergies" of third edition, you could tweak and bend characters to your hearts content. It was about leveling, and gaining power, and being cooler. It was always about being cooler. Those who don't want to role play can enjoy the game as a mathematical abstraction, best performed with software.
Those who want to role-play can see beyond the numbers and attempt to bring life to their characters. Often there is tension and conflict between the two, but that can be fun as well. I've spent as much time writing elaborate backgrounds for my characters (which nobody read) as tweaking the numbers. Also, if it's the only game in town, there is often no single play style for D&D. Because it's usually D&D or nothing else, it becomes the Swiss army knife of role-playing games. Want to run a mystery? Sci-fi like plane jumping game? Espionage? If D&D is what people play, you found a way to make it work. Thus the endless house rules that everyone has for their game. One of the reliefs of D&D 4 is the "fresh install" of a new gaming operating system, the same thing that maddens some people who have purchase hundreds of dollars in "applications."
So there you have it, a combination of play styles, a theme that's abstracted enough from reality, and a giant player base. So why do people love to hate it?
Oh yeah and check out this funny D&D video.