You have this problem because of insufficient overlap. If you fly above buildings, which are logically higher than the ground, you will have less overlap, because your camera is much closer to the building(s). So if you were flying with an overlap of 80% front and e.g. just 50% sidelap, you should go for 80/70 or 80/80%. The more overlap your pictures have, the better results you reach. An alternative solution is to fly higher, but then you will have worse resolution on the ground.