Hi James,
I think your idea is not correct Actually invalid points are points matched in 2 photos or more but not accepted during aligning process for not satisfying certain threshold (far,inaccurate,invisible,weak,,..).
So if you look at an image the grey points are matched points that were rejected during alignment or optimize process. Blue points are points that were accepted and valid... see attachment, where you can see grey points (invalid) are present in at least 2 images...