Hi Deronm,
1) Your flight plan don't look very well, do it like a grid with a high overlap (80%-60%)
2) The uav-terrain distance obviusly is not equal to uav-tower, so take extra photos with a higher elevation of the tower
3) take a look on this site, specially on the "Large Vertical Objects reconstruction": https://support.pix4d.com/hc/en-us/articles/202557459
4) sorry for my english :p
What tutoss said. You need more overlap, and, likely, more photos in general.
I'm not sure what you mean about the metadata of the Inspire1... the EXIF data? Any GPS points likely won't be accurate enough and could potentially hurt rather than help.
I'd add that, if you're trying to make an orthophoto of the
facade of the tower (unclear from your post), top down orthographic photos aren't the way to do it.