Could you please explain a bit more about the calibration process you mention using for this lens? Do you mean you adjusted the mask so the black edges of the circular fish eye are removed? Or do you mean you used Agisoft camera calibration tool?
Yes, this is definitely something I have to write up.... but it goes something like this.
First, the lens calibration tool is not so practical in this case because of the extreme fov.
Secondly, you need to understand what is actually being calculated. The position of each point in an image is being "fixed" by recalculating it's distance from the optical centre of the image using an equation with the lens parameters as constants for the equation. In order to do the reverse, you need to provide images where points in the scene appear in multiple images, in different positions spread from the edge to the centre of the image. With a circular fisheye, this actually requires turning the camera, not just panning.
I prefer brick walls in reference scenes as you can visually assess whether the settings are good or not. Remember that this is a mathematical algorithm... Photoscan doesn't "see" bricks and straight edges. The walls in the point cloud should be flat, with little noise, and the bricks should be rectangular and of consistent size. If you look at a screengrab of the project I used to generate the lens parameters you can see the camera positions I chose to capture the confined space. 1. Walking back and forth through the space, and then a vertical pan with the camera looking across the space from opposite sides. I probably would have done a grid of paths if I were shooting a calibration scene, but this seems to have worked in this case.
So then it's just a case of cycling through the alignment steps to refine the point cloud.
1. Align images
2. Gradual selection and removal of points using Reprojection error = 0.5 and Reconstruction uncertainty = 35 (it's important to do both in this case)
3. Optimise cameras (Do not optimise skew)
4. In the Camera calibration, save the Adjusted settings and load them into the Initial settings (this should chnage it to Auto to Precalibrated)
Repeat this a couple of times. You know you're getting close when the maximum Reprojection error < 2.5 and Reconstruction uncertainty < 150
Once you've got your final settings I tend to Fix the lens parameters for the first alignment of a project to avoid the lens parameters getting re-calculated to values that will throw things out. For testing, go out and shoot a big brick wall from a distance of 2m or less. If the bricks at the outer edges of the model are distorted then you may have some more work to do as this will have an adverse affect on calculating camera positions and create a lot more noise throughout the point cloud.
Full frame fisheyes are a bit easier to work with, and probably wouldn't go below 18mp cameras for circular fisheyes.