Generally there are certain aesthetics in certain types of video/film formats that give off the format it was shot in, and usually you can pick up on them over time. The OTW videos visually look as if they were shot on tape. Also, the director for Rock with You and SOOML told Rolling Stone that they were shot on the same day and for very, very cheap; now given that tape was generally cheaper than film stock, there's also that.
I'm also pretty sure that I read a source some years back that confirmed it was shot on tape, but I'm struggling to find it now.
Could simply be they chose not to, or the fact they didn't have enough time (and they really liked the Human Nature sequence). It's an extremely and overly long retrospective tbh, I tend to skip it most times I watch the film.
Among the time-related reasons for Ghosts not being included, there's also the fact they probably want to save Ghosts full-length for a future release. Why would people re-buy Ghosts if they already have the full-version? Sure some MJ fans would, but many people wouldn't.
As for HIStory, IDK I've never seen it myself. Probably it's because it was to a remix, it also lacked MJ's no presence in the film itself and he probably had little to do with the production (whereas BOTDF was the original music video, just dubbed over with a remix. Plus it actually featured MJ).
As for Smooth Criminal, why would they include half the Moonwalker film? It's the same reason why Michael didn't release the 40 minute version back on the HIStory DVDs. Besides, the rest of the 30 minute segment featured in Moonwalker is ultimately irrelevant to the 10 minute music video, apart from brief appearances from the children and the troopers at the end (who even then don't need the context from Moonwalker, most viewers assume that Michael is playing the Smooth Criminal and the troopers are after him).
They showed the full version of Speed Demon as the context was needed to help show why Michael was being chased in the first place, plus it's only 10 minutes as opposed to a whopping 40 minutes aka half the length of his feature film.
This is a lot more complex than most fans realise. It seems like the only final outputs of Michael's short films exist on tape, even if they were shot in higher quality. Some fans have said they were shot on film then edited on tape. I'm not entirely sure, all I know is that for the most part, the Estate only have the final versions in tape quality. This is why in Bad 25, when we see the outtakes/behind the scenes footage, it's in great quality (as it's scanned from the film), but when we cut to see part of the final product, it's suddenly in low quality. Therefore, it's quite reasonable to assume that the Estate do not have copies of most short films in a quality better than standard definition.
The restoration process for all of Michael's short films would be quite intense and could easily take months and months, even years. Firstly, they would have to find all the original negatives for Michael's short films, and given his sporadic archiving process, this might be harder than we think. On top of this, these have to be the actual negatives of the takes that were selected for the actual final cuts. All of these film negatives will then have to rescanned and touched up frame-by-frame to fix any blemishes/scratches, or even tears if the film was poorly handled/archived. They will have to hunt down EDL's (Editing decision lists, basically lists all the takes used/where they were cut etc) and use those to reconstruct the films from scratch, apply the same colour correction that was used in the original versions. Now with a basic assembly in-tact, they would have to redo all of the visual effects, which got pretty intense from the Bad/Dangerous era onwards. The Dangerous era onwards especially started using digital effects, all of which would've been rendered in standard definition quality so all of the CGI you see in BoW, Scream, Earth Song etc would need to be re-done from scratch (assuming they don't have the original project files, which they may or may not).
So then they have to restore, remix and remaster all the audio from stereo into surround sound (as that's essentially a given nowadays for releases, especially remastered releases). Here's hoping they have all the tapes for the audio files we hear in the short films! There is a process that allows you to get a mono or stereo source and "seperate" what you hear into multiple tracks, but that process is still relatively new, is not the highest quality source possible and probably somewhat expensive, especially for over 30 short films (my educated guess, but the cost aspect is something I'm not entirely sure of). For quality purposes, you would definitely want to go back to the original recordings if you could.
Congratulations! You have one Michael Jackson film in sparkling, crystal-clear high definition. Now repeat this over 30 more times
Hardly. The thing is, Ultimist does an alright job colour-correcting Bad (and I agree, they should re-correct the colour to what Scorsese intended) but apart from that, he does not have access to better quality sources than what the Michael or the Estate have released over the years.
(S)he can upscale/sharpen/use digital-noise-reduction all (s)he likes but it's not really improving the quality, there's no more "information" to utilise in the images (s)he's "improving". In fact, if anything, his/her HIStory teaser looks like shit because (s)he's smudged it to hell and back with DNR, giving it an ugly waxy look (and if anything, you're LOSING detail by doing that).