Hello rowild,
Apart from the confidence filtering (that you can use with factor 1 and 2, for example), you can try to use selection by color option or even try to build the mesh from the depth maps, skipping the point cloud generation stage.
As for the confidence filtering back again, note that after points with the desired confidence value are selected, you can de-select part of it in the Model view by holding SHIFT key and using the the left mouse button to de-select the points from the existing selection.