It's true that Michael would've lost many fans as a result of the allegations. They absolutely destroyed his reputation.
When the first ones hit, some people might've believed them to be true and stopped being fans all-together whereas some might've been unsure and distanced themselves somewhat, feeling uncomfortable. Many people believe that Michael was guilty too as he paid out the family (although anyone who's actually looked into more than a handful of court cases knows that just being you pay out does not always mean you're guilty. It's a case by case basis thing).
By the time the second allegations hit, his name was essentially reduced to a punchline. People were focused more on his behaviours and odd appearances and this had essentially overshadowed his music career, already losing some people as fans.
When your accused of such a disgusting crime again, people get extraordinarily suspicious (fair enough) and distance themselves heavily from the artists work (see Bill Cosby). Additionally, Michael's public behaviour at the time did not help. He continued to allow children to sleep in his bedroom, publicly praising it on television whilst holding hands with unrelated children on camera. Especially on the surface, this behaviour was very alarming to many people as this is simply not normal behaviour from a man in his mid-40s. The media were out in full-force as well so I'd say these second allegations were likely the final nail in the coffin for many fans.
It took for Michael Jackson to die 4 years later for his reputation to begin recovering. It was only then that his name began to recover from being a punchline, that people began to actually focus on the music and realise the insanity of these false allegations against him. Over the years, I've seen more people actually research/read comments online about what actually happened and doubt any guilt they believe Michael had. It's good to see people actually stand up for him in comments now-a-days, public perception of him has changed and for the better. It's sad he had to die for that to happen though.