dowallin, I don't do a lot of aerial work. I mainly shoot and model inanimate objects. But for what it's worth, whenever I shoot anything that has no distinguishable texture, repeating patterns or is moving even slightly, it doesn't work. Meaning, the model doesn't come out as an accurate rendition of the object itself. My main indicator of this is that the cameras are all piled on top of one another after alignment. Then I know that I have a texture issue.
So seeing as the program doesn't know whether it's looking at a forest or a baseball, I'd be willing to bet that what you think is wrong is what is wrong. I think your analysis is correct.
If you're using the professional version you can try placing markers and see if any of them take, that is, get picked up and repeated by the program itself when it recognizes in other photos where you have created a marker in the first photo.
And hopefully someone who has a bit of experience in doing aerial work can give you some more pointers. Someone like stihl or Mehoo.
Jon