Currently seems like after the initial press of the keys the breathiness comes just a few milliseconds before the tone of the note is produced which results in a "laggy" experience.

That "laggy" experience is exactly what the programmer's were going for - this pretty much matches the instrument being emulated - the classic Mellotron - which was an early synth that recreated sounds by using magnetic tape that hung inside the body of the instrument - when you pressed a key it would engage that piece of tape to play. It was "laggy" - that was part of its charm. The use of a BPF give it that smoky, not too clear sound that makes one think Mellotron.

What parts of the voice can I adjust to remedy this?

To edit a VOICE at this level you will need to use the Yamaha S90 XS/S70 XS Editor Standalone/VST on your computer. In order to edit down deep at the individual Element level you will need the Editor. FREE download at
http://download.yamaha.com Both Windows and Mac versions are available

This Voice is made up of 3 Elements:
Element 1 is a concert Flute
Element 2 is ambient noise (yes the Mellotron was analog tape and therefore a bit noisy) this element only adds a low rumble that (again) was part of the Mellotron "charm"
Element 3 is a sample of an AirHose - this gives the whirling sound that gave the Mellotron Flute its character. It comes in very late

The Flute Element's Amplitude Envelope Generator is indeed a bit slow - in the screen shot above Element 1 AEG is shown - the ATTACK is set to 10
Also the KEY FOLLOW is set so that there is this "laggy" feel.
You can adjust the ATTACK TIME to 0 the quickest.
And set the TIME KEY FOLLOW = 0 and set the Center Key = C3

Below is the Amplitude Envelope of Element 3 = you can see it is quite slow

Also it seems to be more pronounced in the lower notes and not as much in the higher ones.

Yes, it will. The lowest possible note on a flute is C3 (middle C) and so any pitches that you hear B2 and below are the sample made at C3 being played below its natural pitch - so yes, it will get very laggy and sluggish. There are no Flute notes to sample below middle C - if you wish to have a responsive sound you might want to edit in a different Flute-like(sic) wave.

The thing is the Flute as an instrument does not have any notes below middle C.

Oh My God, I love you. Thanks for answering so promptly and with awesome detail as usual!