Does Boss claim this pedal is compatible... I'm just saying. The issue is not, in all likelihood, your Montage. Here's what you need to know:
The problem is most likely the sustain pedal and/or a setting. Nothing physically wrong with the Montage

The Montage has several Sustain pedal modes. One for tip-ring-sleeve type connections and the other for tip-sleeve connections. The 'TRS' type is referred to as the "FC3A" after the recommended Yamaha Sustain pedal. The FC3A allows the performer to take advantage of the "half dampering" function, within the AWM2 Amplitude Envelope Generator. The FC3A generates all values 0 through 127. This provides a much more realistic piano type behavior from the sustain pedal. The 'TS' type cable connection is referred to as "FC4A/FC5" after the recommended Foot Switch. This behaves as a simple momentary On/Off, 0 or 127 Sustain function.

Go to "UTILITY"
Touch "Settings" > "Sound"
Find the "Sustain Pedal" parameter
Set as appropriate for your type of cable connection.

Does the Boss pedal generate all values 0 through 127? If not, you must factor that into your selection.

If using a TRS cable, but the Boss doesn't send all values, select "FC3A (Off)
If using a TS cable select "FC4A/FC5"

But, I should mention, getting the recommended pedals, is highly recommended. You got a Rolls Royce, get the good tires!

BM is missing the info. above. The issue is probably with the Boss pedal. Also Yamaha's pedals work in the opposite direction with other manufacturers. Default is "contacted" and depressed is "open" instead of "open" and then "contacted" when depressed. Not sure if that matters in your case.

Polarity doesn't matter for a pedal with a polarity switch.

https://www.manualslib.com/manual/314850/Boss-Fs-5u.html

Uses a 1/4" TS jack - no specification for on and off impedance. One would think this could be done mechanically - but mode switch, battery, etc point towards a transistor - based design with some non-zero (something higher) and non-infinite (something lower) resistance.

Batteries (voltage) may affect the results - and if the impedance is marginal vs. what Montage expects - then even temperature can make the difference between working and not working reliably.

Sustain tends to be one of the lesser expensive options. $30 for a direct replacement of the FS-5U (Yamaha FC-4A). If you want to extend the function of the damper beyond what the FS-5U could ever do (even if it worked) would be to get a Yamaha FC-3A - in which case you'll spend about $13 more for that feature. Or add $11 (to $30) more for open box - not really worth the $2 savings in my opinion.

Thank you all for your quick replies.

As Jason states: polarity is not my problem, the pedal has a polarity switch. Seems like Bad Mister hit the point: my sustain pedal was set to FC3A (half on). I changed it to FC4A / FC5 now and will let you know if this solves my problem.

If it doesn't work it wouldn't be to much of a problem to exchange my pedal though. I just like the FS-5U because it has a "rail" to connect it to another FS-5U (which I need for my other board on stage). This is super practical for life use, because the pedals stick together and stay exactly where I want them to be.

Just to let you all know: after adjusting my sustain pedal setting according to BM's instructions I've been playing a couple of live gigs in the last four weeks without any problems. Thank you very much!