You convert SysEx files from supported FM keyboards using the FM converter online here (under the APPS) section and then load the library file it generates.

https://www.yamaha.com/us/downloads/FM-Converter-User-Guide-English.pdf

So if I find an FM library on line, I can load the library without any conversion?  

Maybe Yamaha engineers should talk to Kurzweil engineers.  Kurz keyboards can load SYX files directly.

A midi app (e.g. MIDI-OX) can load a SysEx file but only if that file is in the proper format for your instrument.

Is there an instruction set to loading FM libraries?

If by 'FM libraries' you mean a library file produced by a Yamaha instrument like Modx, Modx+ or Montage then, yes, the manual for your instrument has instructions for loading libraries in the 'Utility-Contents' section.

The instrument itself can NOT load SysEx files directly though there have been past IdeaScale submissions asking for that functionality to be added.

If you find content made for recent Yamaha synths (see manual for compatible formats) then sure you can load those directly.  There's a million (give or take) DX7 or other FM synth patches in SysEx format in the public domain.  But if you can find what you need in a library or user file for Montage/MODX/Motif then go with that.

Korg loads SysEx directly too so Yamaha has been been able to see competition doing this up and down.  The file system, in general, has been a weakness for a while.  I'm not sure making things streamlined has been top priority.