Hello Lupus,
I don't recommend to remove too much points from the sparse cloud, because having too small number of matching points between image pairs will lead to skipping such pairs in the depth maps generation process, so that the dense cloud may be inaccurate or have holes in it.