Hi sbond,
it COULD be about the points situation in space. Every point has x,y,z-values for its position and when you alter the angle of your model, the point might not be visible from that perspective.
Have you already tried to edit those "noise points" in the Tie Points and build the Point Cloud with the clean data? For this step, i am having pretty good results with the "Gradual selection" Tool which becomes available from the Model Drop-down menu once you have the Tie Points active. Here you can select points by reprojection error and reconstruction uncertainty with the help of a slider.
Good luck!