Thanks for the question… Hopefully, we can give you some things to try that may help get the results you desire.

The Vocoder can be, as you observe, tricky to use in a setting where there are external sounds getting into your microphone (leakage). Ideally, isolation of the source (your vocal) should be optimized, and as with all microphone/speaker systems, the sound coming out of the speaker must be prevented from re-entering the microphone (to avoid feedback).

Howling, and the high frequency sibilance so bright you need sunglasses to be in the same room… these are common issues when using the Vocoder under less than ideal conditions. What follows are some solutions… and an explanation of what and why...

First thing to understand about the Vocoder, it is not an Effect for the microphone. Thinking like that will lead you in the wrong direction. The Vocoder is applied to the synth Tone Generator, the microphone is a modifier. The signal coming in the microphone is applied to the synth Part as a modifier. The signal from the A/D IN is routed through the synth engine instead of to the Main L/R Output, as would be the case normally. The signal from the microphone enters the synth via the synth Part’s Insertion Effect, and is passed through a set of special frequency band filters… the resulting signal is encoded onto the synth sound.

Vocoder = vocal encoder

What you usually expect from the microphone is a musical combination of musical tones and noise that we identify as someone singing/speech.
Vowel sounds can be given pitch, while consonant sound are generally, non-pitched components.

Quickly, if you sing the words “smooth” and “squeeze”… it is the vowel sounds, “oo” and “ee”, respectively, that you can apply a pitch to… the sizzling “s” sound that starts both words has a rather fixed pitch in both cases, fixed for each performer. You cannot change the pitch of this consonant noise component as freely and easily as you do for vowel sounds.

The vocoder's filter system does the pitch removal - what gets passed on to the synth engine are your noise components… it is these (consonant) noise components that give clarity and intelligence to the vowel tones. The pitch portion of the incoming signal is replaced by the pitches fed from the keys being pressed.

Long story shortened: What you need from your Vocoder-mic is not what you normally look for from a microphone. I, too, have a Shure Beta, which is excellent being a dynamic Supercardioid (unidirectional) microphone… I've traveled with one in my computer bag so that, when necessary, I could quickly demonstrate how the vocoder worked. For teaching, clinics, and the like, this was ideal. I could demonstrate how to get normal signal through the keyboard, and also demonstrate the vocoder.

Well, under the ideal conditions where I was the only audio signal happening in the room, I could do a fairly good demonstration of both use cases, normal and vocoded examples. But if I was on-stage with booming bass, drums with cymbals ringing, guitars, etc., etc., it would not be such a good situation for the vocal demonstration.

You often see touring musicians with two microphones — now you know how they got here. One for singing and talking normally, the other for the vocoder. The vocoder-mic does not have to be an expensive mic… This is because we are going to remove all of the low and mid-range sounds it does so well… You’ll want to be able to get really, really close to this mic, called “eating the mic”… because you want it not to be able to pick up anything else. So you will optimize the level so that when you are at full throat, the mic is just picking you up. This will help those other sounds (leakage) at a minimum.

Most users test the mic level at normal talking volume, but don’t realize that when performing with the vocoder function, you are typically at a much different output level. So first thing is to learn to set the Gain for your voice when “vocoding” — full throat - this gain staging is critical. Get this wrong, it can become a disaster.

NOISE GATE possibility:
Fact: the Insertion Effects you put on the AD IN are before the signal goes to the Synth Part that has the Vocoder assigned.
You can applied the “Noise Gate/Compressor/EQ” to the microphone as an Insertion Effect… this will help a bunch. But comes with a "learning curve". You want the NOISE GATE to ‘open’ to allow your vocal through, but ‘close’ on the unwanted sounds that are leaking into the microphone. This “Threshold” level setting is critical, so you want to set it for your performing level. You want the COMPRESSOR to give your vocal signal presence. You will need to experiment with the EQ… I say that because, what follows this Insertion Effect is the signal gets routed to the synth Part, where the Vocoder and its elaborate set of frequency specific filters are applied… the object is not to EQ the microphone signal the same way you would normally. In fact, when you work with it a while you’ll discover you are in a different space - leave it flat at first… work with the Vocoder parameters within the Synth Part’s Insertion Effect block and only use AD IN’s Insert Effect EQ to make corrections after you hear your vocoded signal.

Alternative Solutions: That actually work…The "DJ Trick"
Although you will not find this in any textbook, but necessity is the mother of invention….
We understand that signal leaking into our microphone makes for a bad vocoder sound. The extreme attention to Gain staging the mic for very close proximity vocals, the Noise Gate/Compressor combination on the AD IN signal, and physically positioning yourself and the mic as to avoid the feedback possibility, can all help, but what is ideal would be complete separation of the microphone from any outside sounds. A ‘vocal booth’ for your mouth and the mic!

Microphones and Speakers are the opposite of each other. They are both “transducers”… one takes air molecules in motion and converts them to electrons; while the other takes electrons and converts them into air molecules in motion. What this means is: You can use a speaker as a microphone!

With this knowledge find a pair of headphones with nice large closed-ear ear coverings.
Plug it into the L/Mono AD IN, (make sure you MIC/LINE = MIC), talk into the Left earpiece. (The other ear-piece will not work).
It works as a microphone, with its own isolation chamber. oh yeah, it looks strange at first, but the quality of the results versus those under the typical on-stage sound leakage conditions, will speak for themselves.

For use with the Vocoder (which does not benefit at all from a high quality microphone with stellar frequency specs — because a vocoder is filtering mostly all of pitched frequencies anyway) the headphone as microphone trick works perfectly. It should be able to capture your vocalizations and apply what is needed to the Vocoder.

There are even headphones made with a single ear-piece - used by DJs and violin/viola players when doing studio work… but an old set of headphones can work great to isolate your voice. Your mileage will vary… I have a set with a nice large ear cup, that stays in place when I put the other ear-piece so it is on the back of my neck… looks strange, but the vocoder sounds clear and free from a vast majority of leakage from the band.

Also - the 58 as an omnidirectional mic is great at picking up off-axis sound.

Hypercardiod would be better if you can make one compatible.

Do I need a headset mic that will put my mic much closer to my mouth or is there a way to gate the input on the vocoder part? I'm currently using a beta 58 and just getting frustrated trying to bring this part into the song live.

I bought a Shure WH20 headset mic with the windscreen just for the vocoder effect only. And I "Eat the Mic"... I use a separate mic for lead vocals.
I also use the vocoder effect with choir voices and special EQ'ing to do harmonies with my lead vocal in some songs, so I basically eat the WH20 windscreen, but stay about 5-7cm back from the main vocal mic.

Thanks Bad Mister for such a thorough explanation of the subject. I think I should be able to get things working with all of that information. Thanks also for the suggestion on the Shure WH20 Darryl.

Jason, the Beta58 is a supercardioid mic, not an omnidirectional, but a hypercardioid would be better. I still need a seperate mic for backing vocals so I expect I'll use the 58 for that and look at something like the WH20 Daryl suggested or a hypercardioid of some sort like an Audix OM3 or F5 for the vocoder.

Hey @Dan,

Just a note on the Shure WH20. I had done research on the proper mic for the Montage and this headset mic is the right match, as it is a Dynamic Cardioid (unidirectional) mic.

There are 3 different ends you can get them in, although the 3rd is to plug into a Shure wireless pack, so here are the main two:

Shure WH20XLR (XLR end)
Shure WH20QTR (Right Angle 1/4" jack)

Note that the cable is quite short (4 feet), so you may consider an extender cable to go with it if you do decide on this particular mic. I bought the Shure WH20QTR