Hello airmap,
seems like you have built geometry based on point cloud??? Or low quality?? Try to build geometry on high.
Another option is, to cut the building out of the model --> close Holes, than you can choose "one" photo that contains the whole structure in a nearly orthographic projection. The scale will be correct only in terrain region. Do not delete to much of terrain, since "close hole" geometry have to be aprox the same level of the real terrain situation.
Best regards