There were certain unresolved rifts and tensions among the Jackson brothers that played a big role in that.
For example, during the Victory Tour Jermaine Jackson appeared to be extremely jealous of Michael Jackson.
Jermaine Jackson was bragging about being the hottest of the Jackson brothers, a behavior that Michael Jackson did not like at all.
Also, although there was a written agreement that the Jackson brothers themselves were only allowed to ride in the van chartered to take them to concerts, Michael Jackson began taking in the van his close friend Emmanuel Lewis along with them.
The rifts and tensions were so intense that at one point (during the Victory Tour) Michael Jackson abruptly left his brothers and he went to Disney World (August, 1984) in order to clear his head.