Nope, I take it back, there is something strange.
Every so often, and more often than I'd like the mask is the exact opposite of what it should be.
I think my camera positions are good? The geometry is nice and clean and all other things (placing markers etc) appear to work as expected. But not the generate masks.
Is this just a mistake in my camera alignment?