Is there a way to "process" audio output of the ModX directly while playing midi?

I suppose that the "professional way" is: record audio tracks from MIDI of each single instruments, and process audio tracks from cubase

Yes, there is a way to process your audio output from the MODX directly. The way you have described is one "professional way" to do things, but it is certainly not the only method.

Recording MIDI is done because it allows you to change your mind. While your musical performance is encoded as MIDI Data, you still have an opportunity to easily correct wrong or unfortunately placed Notes, change the instrument sound you are triggering, create music charts using the SCORE feature, etc., etc.

Once those correction are made you can render your performance as MIDI. This entails playing back the MIDI Track to the target Tone Generator, and capturing the AUDIO OUTPUT. This method is popular (but no more or less professional than others), because you do not have to deal with latency. Latency occurs when you ask the computer to receive audio signal, process it with plugins and/or print it to the hard drive and send it back to the main audio outputs (usually a few milliseconds of time for all that processing).

In the "MIDI Record - then render Audio" workflow: the 'rendering' of audio is done by the computer in a process where you are not necessarily listening to the data as it is converted into an AUDIO WAVEFORM. It uses the time stamp functions to accurately place all the data where it belongs. In Cubase, this is called EXPORT > AUDIO MIXDOWN

But if you want to process the audio that is coming from the target MIDI tone generator (in real-time), and you wish to also monitor it... there are other ways to configure your setup. Example, you are processing your MODX with a plugin that alters the sound... it is most convenient to HEAR that processing while your playing. No doubt a musician will play differently when playing with the processing - if it is a guitar sound, and you are using a plugin to create Distortion, Compression, and Delay via plugins, you will, naturally, want to HEAR what you are doing while you are playing it. In order to do this on today's computers means you must deal with the small amount of computer processing latency introduced by the system.

While Direct Monitoring = zero latency...
Recording while processing Audio will mean you are monitoring it after the signal has been processed (and slightly delayed) in the computer. Don't panic - if your latency is less than 10 or 15ms, you will hardly notice. In general, mileage varies by your computer hardware.

In your case, you want to send the audio output to the plugin and Monitor the signal post (after) the plugin. This way you get to hear what you are doing in that processor. This is another way of setting up to record.

When you go to [UTILITY] > "Settings" > "Audio I/O" > Turn the "Direct Monitor" = OFF

What this does: it disconnects the MODX Tone Generator from going to your MODX Analog Outputs (both the Main L/R Outputs on the back panel and your PHONES output _ which are always the same signal). In the AUDIO SIGNAL FLOW diagram (shown in detail below) you can see the yellow line starts at the MODX "Part" and can either travel on the Main L/R (USB) Output or it can be assigned to the USB1-8 Outputs, on a per Part basis.

Yellow is signal in the MODX before it goes to the computer... Blue is the signal returning to the MODX now acting as audio interface, which converts and sends analog audio to your Main L/R (analog outputs).

You will no longer be able to hear the MODX, unless/until you route the audio signal (now digital) through the DAW (Cubase)... and you activate the Monitor icon on the Audio Track.

Create an AUDIO TRACK in Cubase, set to receive audio from the MODX - activate the speaker icon to monitor the sound
You can choose the Main L/R (USB) or you can select any of the USB1-8 Outputs either in odd/even stereo pairs or as individual mono sends to the computer.

The Audio Track in Cubase will allow you to INSERT whatever plugins you want in the path of the signal.
You will now be able to monitor the signal post the audio track... What you hear is what will be recorded to that Audio Track.

For a detailed tutorial on getting into AUDIO RECORD please see the following:

Link -- Mastering MODX: AUDIO RECORD ON DAW Part 1

Link -- Mastering MODX: AUDIO RECORD ON DAW Part 2

If you get stuck or have more questions, post back here.
You mention you are using Cubase 11 but not which model_ reminds us to mention if you are using Cubase Pro 11, you have even more options for setting up than are available in Cubase AI 11, for example.

- Turned OFF "Direct Monitor"
- created audio track in Cubase (version 11 AI)
- activated monitor icon
- now I can apply effects in real time
It works! Thanks 🙂