First, RPG sales are fine. For us they were down around 10% last year, but that can be attributed almost entirely to the twilight days of D&D 3.5. Many, many people (millions) play role-playing games and although they tend to be older, younger people are getting involved too.
Second, the market is very mature. In mature markets, the winner tends to take all or at least around half. Look at something like soft drinks. Coke has around 50% market share and the second 50% is spread around among everyone else. That second tier struggles to survive. Wizards of the Coast is the Coke of RPGs. In my store, no other game has more than 10% of sales.
Third, the pie is cut into too many pieces. Wizards of the Coast maintains their 50%, but the other 50% is spread out very, very, thinly among mid-tier publishers, small press publishers, and everything available via PDFs.
With PDF's there is this concept of the "long tail" in which RPG games will never die, they just get distributed electronically. On RPOL people favor old editions of games, like Shadowrun 3E. Now imagine a market in which a relatively small group of people can choose among 30 years of published products. It's great for them, but it spreads the money so thin that only the biggest companies can survive.
Why would Games Workshop close down a division and drop a relatively solid RPG product? Because it's spare change to them. They come from the miniatures hobby, which is much healthier and less fragmented than RPGs.