I figured out a solution that works for me.
I create a group of all the inputs that I want to loop.
That group has its output set to sends only.
I set monitoring in for all of these.
The input for my channels with loopers is the name of that group.
I then create another copy of of that group for recording purposes, on that group, the tracks are armed and everything is set to monitor off. Basically the monitor is required to send to the looper but we don’t actually need to hear it if we are careful.