Why dont you filter by confidense, then select all the remaining points and assign then a class. After you reset your filter you can still see all the points, however during DEM or Ortho construction you can delegate which class to base the products off...i do it all the time