This isn’t a trivial problem to solve in software and will always be YMMV. What you have to understand is that despite it saying your devices are both 48kHz, they’re actually not. They’re either running a bit slow (think 47.982kHz or something like that) or a bit fast (like 48.005kHz), so one will always take the data out of your buffer a bit faster than the other. If this is not compensated by duplicating samples or upsampling, then downsampling twice (with some kind of PLL in between), the buffer is going to run out at some point for one of the outputs.

Pipewire is likely trying to compensate for that as described, but that isn’t free (more than double the work it has to do on a single output) and when something takes a lot of CPU time this can starve Pipewire out for long enough so it can’t get this work done before it’s due (so to speak). The quantum you’ve read about is essentially the size of that buffer I mentioned above and the bigger you make it, the more slack Pipewire has to work with before the buffers get depleted. This translates directly to a latency in ms if you divide this number by the samples per second (i.e. the frequency), so 2048 like below would translate into 42ms at 48kHz. Note that your sound device will have its own internal buffer that also adds latency on top of that.

I’ve had the following in my Pipewire config for years at this point because I had constant crackling in my (single) output when the CPU was under load:

context.properties = {
    default.clock.min-quantum = 2048
}

I’m not sure if this is still the suggested way to configure this, but it has worked well for me over the years. (Or maybe it’s dead configuration and the problem went away on its own separately, who knows… maybe time for me to look at this again.)

PS: No idea how the buffer running out translates into the shift in pitch you observe, so I can only speculate. Your hardware, ALSA or Pipwire itself might try to compensate by switching to a slower sample rate without resampling the input data to avoid the buffer underrun. I.e. one output is playing the same data, but slower to allow the other one to catch up. The fix might still be to make the buffer larger. Another option would be to make Pipewire have more priority, but since it’s a user process it can’t go below its default niceness of -11. Your only option would be to make everything else more “nice”, i.e. games, browser etc., but IMHO that’s more trouble than it’s worth, you’d better be off to find a quantum value that works for you.