Personally, I don’t see this as a lack of interest/drive as much as MJ embracing his own inconsistent creative process.
Quincy was a quick, efficient producer: when sessions started, he knew what songs he wanted to prioritize, and he turned in a finished album within a couple months. I don’t think it’s a coincidence that this was the period where MJ was putting vocals on nearly everything he wrote: half-finished song sketches were less likely to be selected than mostly-completed demos, especially when MJ was still going up against Quincy’s team of songwriters.
Once MJ took the reins of executive producer in 1989, that structure and focus was gone, for better or worse. Suddenly he was more willing to let songs gestate on their own time, which meant not getting in the booth until he felt it necessary. He was still churning out dozens of new songs per album/project, but he wasn’t under the obligation to impress anyone but himself.
I’m sure age and his passion for an acting career played a part, especially by the late 90s and early 2000s. But I chalk his shift away from vocal recording to not having the structure and sense of competition Quincy provided.
I believe that Quincy might have played a part in that, and the competition with people like Rod Temperton certainly made Michael’s perfectionism even higher, but I’m not sure it’s the full picture. I recall John Barnes talking about how after Thriller, Michael thought that his collaboration with Quincy had finished and that their 3-project contract had been fulfilled, with the release of the E.T. soundtrack album.
Now I don’t know exactly when it became clear to Michael that that wasn’t the case, but Barnes has stated that when he came in for the first session at Hayvenhurst, the consensus was that they were working on Michael’s upcoming album. He is also extremely disappointed in people calling it the “B-Team”, because according to Barnes, the album was originally supposed to be produced by them. This is the original quote from him:
“Just to be clear, we didn’t go to Michael’s house to do demos to hand over, we went to make the record.”
It seems that after Thriller, even with the uncertainty of whether Quincy would be involved in the next project or not, Michael’s main goal was to top the sales of the album That, I find, explains the constant creative drive and ambition more than anything else. It would also explain why this era seems to have the most “finished” demos.
For Dangerous, even though Q was not involved, Michael was still developing music in a similar way to before. There might be slightly more rough ideas, but overall that era was also highly productive - a good number of songs with Bryan Loren, Bill Bottrell and later Riley. Many of the best ones developed in the early sessions in 89’. And many demos from this era would later make it onto HIStory and Blood, which attests to the quality and quantity of material being produced.
I believe with Dangerous, Michael still believed he could top Thriller. I remember reading that his goal was for it to sell 100 million (same as with Bad). With HIStory and Invincible, the story is a bit different, since they came after the allegations and after two very successful albums, which were still no Thrillers in terms of sales. So, I believe at some point, it became clear to him that Thriller was a once-in-a-lifetime phenomenon that could not be replicated. This, along with Michael’s passion for exploring different artistic avenues, and some of his health issues is mainly what made him be less consistently productive, in my view.