Sondre,

Same problem. Tech support at Yamaha helped me put 4 performances on the master mode. Yes, takes at least 1 second to switch. I used to have a DGX 650 without these problems. Plus on the KORG Krome there is a way to flow one song to another. The only thing I have is on one of the comments from DSetto, suggests using the mixing mode. But I haven't explored it yet. What I am finding is the MOTIF XF is overwhelmingly difficult to learn.

Lets say that I have a sound from voice mode in "1" and a sound from performance in "2". (This is in master mode). When I now change from 1 to 2, it takes an hilarious amount of time, probably about 1-2 seconds for the screen and sound to change. It's a little faster when going from voice to voice, or performance to performance, but damn, so slow! How come?

Because you are switching modes. That is, the "how come". There is never a satisfactory answer to a "how come" question. So there it is, I'm sure you're not satisfied. We do have a solution for you below.

Done anything wrong?

If it takes an hilarious amount of time, but your laughter is not enough to fill the time, yes, you may have selected the wrong mode to accomplish what you need to do.

Do I need to factory reset?

No, what you report seems feasible and quite normal actually... although I never saw the humor in it. I guess I don't have any sense of what's really funny! 🙂

I honestly hate it, and I checked in a XS keyboard, and that worked much smoother all of it. Damn! 🙁

Then let us suggest a workaround that will allow you to recall your setups as fast as you can't touch the buttons, no delay, no cut off in sound, seamless transitions between single sounds, and your multiple layers, splits etc.

First, since you expressed how much you "hate it" and expressed interest in the "how come"... forgive our length in this explanation:
It's really the complexity of each program that causes the delay when moving from program to program. In VOICE mode each Voice has two Insertion Effects, two System Effects, a Master Effect, uniquely setup for each Voice. When you have an inexpensive keyboard DGX, for example, every program goes through the same effects processor setup... less complex, less flexibility, but quicker to recall. Make sense? I know, not really, the reason is never really satisfying...

Move to Performance, you now have four Parts, each of those four Voices has its own dual Insertion Effects, and each now has a Send to the two System Effects (shared by all Parts). Recalling eight Insertion Effects takes longer than recalling two.

Move to Song/Pattern Mixing, you now have sixteen Parts, (this takes a bit longer even to recall - but once you do you have recalled a multitude of resources!) you can now freely set MIDI channels, you can now have eight Parts each recall their own dual Insertion Effects... and because all of the resources are recalled at once, you now seamlessly move between playing any single or any combinations of sounds you can dream up. Up to the limit of 16 total...
.
Under Track button [1] you could recall Parts 1, 2, 3, and 4 assigned MIDI Channel 1 of the MIXING.
Under Track button [2] you could be playing a single Voice assigned in Part 5 of your MIXING, but on MIDI channel 2
All Parts will have their dual Insertion Effects and you can seamlessly move between the single Voice Part and your "Performance" setup copied into the MIXING by simply toggling back and forth between buttons [2] and [1].

No audio cut off, no delay, "how come"? because all the Effect resources are recalled when you recall the Song Mixing or Patter Mixing setup. There will be no interruption in audio because you do not have to interrupt the complex routing - it's all already in place. The Mixing set recalls 16 Insertion Effects (8 Parts each with 2), the System Effects (Reverb, Chorus), the Master Effect, the Master EQ and it recalls them all at one time. Moving around WITHIN this setup is as fast as lightning. You cannot out speed the change between programs because it is seamless. You can even hold one, either with hand or a Sustain pedal, switch to the other and play, continuing holding the first, until you let go of the keys or the Release Sustain pedal.

Some times the solution is just using the right mode for the job. MIXING mode, let's do this.
If you have already setup your Performance, go to a blank MIXING and COPY the Performance to Parts 1-4 of the MIXING. Place the single Voice in Part 5, set the Receive Channel for thus Part to 2.

Hope that helps.