First, if it's not already obvious, back in the day, the late seventies and early eighties for me, there were a lot more people playing. D&D especially had a player base that was probably ten times bigger than it is today, and when I got a purple D&D box set for Christmas one year, it was because it was THE hot gift, not a staple of of a nerdy niche hobby. A lot of us stayed with the hobby long after the fad faded, but the player base remained pretty big throughout my childhood. As an aside, my childhood after moving to California was pretty miserable until I began making friends in middle-school through gaming. It was sports for nerds.
Second, the game was incredibly subjective. I recall going to a couple Orccons back in the mid-eighties and playing D&D in the open gaming room . The place was absolutely packed with D&D games, ranging from simple pen and paper affairs to foam cut out dungeons built up several feet high. When you sat down at the table, you didn't know what to expect. House rules? All rules were house rules, as the game rules, rules as written (RAW), were more a suggestion than a codified map of play. Gary Gygax openly told DMs in the DMG that players could go screw themselves if they didn't like it (more or less). It's not that people were taking liberty with the rules, it was that they were open to so much interpretation that the game experience varied dramatically based on who ran the game.
At one point I recall signing up for an official tournament game, to get a feel for what the real rules were like. They were no less subjective, and superimposed was a ridiculous scoring overlay that tried to give points based on arbitrary actions. Oh, you searched the wizards beard for the missing key? Great, one point for you and an extra day of rations. How clever you are. It was not fun. That said, my favorite adventure of all time is Hidden Shrine of Tamoachan, a tournament adventure that forces the party to race through a dungeon before the poisoned air kills them. Good times.
What the game needed, what it required, was a certain type of person to run it. That person had a basket of skills, including organizational skills, basic social skills, a ton of creativity and most importantly, the ability to tell a good story. There weren't books on how to do this, so there was a lot of fumbling around trying to figure this out. There were way more really bad games than good ones and we made every mistake, including the ones that pissed off our parents and teachers.
A general grasp of the rules was important, but that wasn't so hard. It was not uncommon back then to know all the rules, and possibly even what page they were on. We had three "core" books for quite a few years and we would often try to outgeek each other by showing our mastery of those tomes. We could flip to certain sections by touch, as the books began to wear (usually combat and treasure tables). It reminds me of young students of Tibetan Buddhism whose spiritual practice is to study and memorize one text. Knowing the rules was not about who had the most money to buy stuff, and perhaps that's why everyone took a turn at DMing. Nowadays it's about the guy who can afford to buy all the new releases, and the players who leach off him. Honestly, if you want to get a vision of what that looked like, think modern day Pathfinder, rather than D&D. Fewer rulebooks, but lots of adventures. TSR would screw that up later.
Like painting miniatures, DMing is craft, not art, and anyone can learn to do it. Nevertheless, some did way better than others, taking the bare bones framework of the rules and spinning a mesmerizing story around it. Within my own game group, I freely admit that I was not one of these legendary story tellers. We had several good ones. Jim excelled at modern games, spinning spy tales with clever villains like Taskmaster in his black Porsche 911. That guy was always one step ahead of us. Stefan was a genius, and I recall eating up his game sessions until he inevitably lost interest not far in. Russell would tell the best stories, the writer of the group, although his game design wasn't as good as some of the others. I had a secret DM at one of the city rec centers, an amazing old guy whose name I can't recall (a college student) who had written a giant, multi level dungeon that we had no hope of ever finishing. He had a grasp of the natural world that blew my mind, with vast underground cave complexes filled with rivers and lakes. I kept that game to myself, a prized treasure every Wednesday afternoon. Despite obvious skill in some, it was still craft, and we were all learning. Anyone could do it and over time you got better.
That sense of craft is what kept me in the game long after my friends moved on, running D&D games in dot-com conference rooms and later at my house with wait lists of players, not because I was a DMing genius, but because I was seriously invested in the game when there were few people like that left. Again, I will freely admit not being great at it, but that's what having a hobby is about. I'm a professional game store owner, not a professional dungeon master. I reserve the right to suck at my hobby, but endeavor to improve.
Modern D&D takes a lot of the subjectivity out of the game, but it still requires the same set of skills to run a successful game. It's easy to forget that when your game prep is now divided between story and drawing battlemaps and painting miniatures. The story telling can take a back seat to the energy and effort of running tactical combat. Regardless, a good game master brings it all together and any game system can be the platform for a fantastic game, provided you've got a good game master.