Agisoft Metashape

Agisoft Metashape => General => Topic started by: bigben on January 20, 2015, 04:08:58 AM

Title: Taking fisheye to the extreme
Post by: bigben on January 20, 2015, 04:08:58 AM
Excited by my latest quick test of a combination I should have tried at the start.  Here's a teaser.
https://skfb.ly/CDWo (https://skfb.ly/CDWo)

Shooting a more complete test this afternoon.

[update]Adding more samples here for those seeing this topic for the first time:

Test 1: repeated with optimised lens parameters
https://skfb.ly/CLRQ (https://skfb.ly/CLRQ)

Test 2: https://skfb.ly/CPpn (https://skfb.ly/CPpn)
Entire building and surrounds
Full point cloud of this test: http://files.digitisation.unimelb.edu.au/potree/pointclouds/gatehouse_8mm.html (http://files.digitisation.unimelb.edu.au/potree/pointclouds/gatehouse_8mm.html)
Title: Re: Taking fisheye to the extreme
Post by: bigben on January 22, 2015, 12:56:52 AM
As a panoramic photographer I work a lot with fisheye lenses.  I always had an idea to try a circular fisheye and now I don't know why I waited so long.  The first scene was just a quick series of 21MP  images shot while walking along taking a shot every metre or so... at arm's length from the building, camera pointing forwards.

This screengrab is the sparse cloud of the reshoot later that afternoon.  A previous test with a 24mm lens is here: https://skfb.ly/ArE9  Note the hole in the verandah. I took a short step ladder to get my hand above the flat roof on the annexe of the building.

Normally you wouldn't shoot in the direction of motion as it doesn't provide much parallax, but with a circular fisheye, a lot of the image is looking off to the side, so this doesn't apply.  There is, however, always a part of the image that is in looking parallel to the direction of motion. What impact this may have I don't know yet.

I crop my images to square now to save space, and have a circular mask image to mask out the edges.

So if you own a circular fisheye lens start by trying this... Go to a narrow laneway/street, walk along taking an image every metre or so, turn around and walk back taking more images. 

My test building also has a small alcove on one side.  I've shot this with a 15mm lens and it worked well, but it worked better with the 8mm, taking less images and not creating an area of excessive point density.  At longer distances from the camera, noise increases quite a bit but it's great for working in the 0-5m working distance range

Revising my shooting strategy with more tests.
Title: Re: Taking fisheye to the extreme
Post by: Kiesel on January 22, 2015, 01:16:25 AM
Indeed very nice results from a fisheye lens! Thank you for posting.

Karsten
Title: Re: Taking fisheye to the extreme
Post by: bigben on January 22, 2015, 03:36:21 AM
One more screengrab with the cameras visible and some thumbnails.  I was testing a few ideas to see what would and wouldn't work, including underneath the picnic table  ;)  The other thing that may be of interest... this is about an hour's worth of shooting.... and for reference, this is the location: https://www.google.com.au/maps/@-37.799871,144.9604602,169m/data=!3m1!1e3?hl=en (https://www.google.com.au/maps/@-37.799871,144.9604602,169m/data=!3m1!1e3?hl=en)

I'll generate a dense cloud over the weekend and put up some models of different sections for evaluation. 

Currently running a dense cloud on my other test subject... the atrium of the Melbourne School of Design.
http://thedesignfiles.net/2015/01/melbourne-school-of-design/ (http://thedesignfiles.net/2015/01/melbourne-school-of-design/)
Title: Re: Taking fisheye to the extreme
Post by: ThomasVD on January 22, 2015, 04:29:18 PM
Would be a good procedure for modeling tunnels as well.. just walk through in both directions with a fisheye cam and voila.
Anyone up for modeling the Paris Catacombs ;) ?
Title: Re: Taking fisheye to the extreme
Post by: bigben on January 22, 2015, 04:46:33 PM
Would be a good procedure for modeling tunnels as well.. just walk through in both directions with a fisheye cam and voila.
Anyone up for modeling the Paris Catacombs ;) ?

I was thinking of different configurations of multiple cameras on a pole. In this case, you'd only need two cameras, one pointing forward and one back. Might be easier for tunnels to have a hoop around your body supported by straps over the shoulder.  Then you only have to walk through once.
Title: Re: Taking fisheye to the extreme
Post by: bigben on January 24, 2015, 12:05:55 AM
Here's a quick (15 minutes shooting time) test of a graffiti-filled laneway.  I failed to consider that the main parts of the image that are used for the reconstruction with this camera orientation are the outer portions. To get better overlap in these parts I should have reduced the distance between shots. 

With a relatively flat ground there wouldn't  be much need to shoot in the reverse direction for the ground close to the camera. Taking two shots at 45° to the camera path would provide more overlap at the sides as these images would also include 45° behind the photographer.  There are a few other practical considerations.... more testing to do.
Title: Re: Taking fisheye to the extreme
Post by: ThomasVD on January 24, 2015, 11:58:24 AM
Looking good, thanks for the updates :)
Title: Re: Taking fisheye to the extreme
Post by: bigben on January 24, 2015, 01:16:40 PM
Put the MSD atrium (high) dense cloud online using Potree.  http://files.digitisation.unimelb.edu.au/potree/pointclouds/msd-atrium.html (http://files.digitisation.unimelb.edu.au/potree/pointclouds/msd-atrium.html)
Some quick nav tips... mouse scroll wheel moves backwards/forwards. Right mouse button pans, left button rotates. It gets interesting once you move into the middle of the model

This was about an hours shooting with no really structured plan, more of a suck it and see.  Camera path was a grid over the floor and along the stairs and balconies, camera pointing in the direction of travel. Bits have worked well, others haven't.   It was a bit ambitious to have such a large space work with an untested idea, but it was a very informative exercise.  I found it very hard to stop shooting and even did a single camera path down the stairs to the floor below.

Running the dense cloud for the gatehouse shoot at max density. I'll probably do my next test with multiple cameras on the gatehouse before returning to the atrium.
Title: Re: Taking fisheye to the extreme
Post by: bigben on January 26, 2015, 02:31:48 AM
Found one glitch that I think is caused by having the direction of travel within the frame.  The light coloured points circled in the image are along the camera path when I was walking towards the large window, which was overexposed. Masking out the windows would have been one options, although a bit fiddly.  Also thought it might be a good idea to mask out a small circle from the centre of the image, thus masking out the area of image with very little parallax change between images.
Title: Re: Taking fisheye to the extreme
Post by: ThomasVD on January 26, 2015, 12:38:50 PM
I suppose glass surfaces always remain difficult. You could try mounting a polarising filter on the camera? http://en.wikipedia.org/wiki/File:Polarizer_Through_Glass.jpg
Title: Re: Taking fisheye to the extreme
Post by: bigben on January 26, 2015, 04:52:45 PM
I wasn't expecting to get the outside of the building as I'd let it blow out.  In the previous screengrab you'll notice a large hole in the wall that is a window.  The desks and computers behind it showed up pretty well as the reflections weren't too bad.  With a circular fisheye you'd lose too much fov by putting a filter in front.  ... The extra fov also poses an additional problem in that the polariser will only be effective for removing reflections at a certain angle and you lens is now looking a a much wider range of angles.
Title: Re: Taking fisheye to the extreme
Post by: bigben on January 27, 2015, 02:55:58 PM
Still waiting for the dense cloud of the gatehouse and preparing some images of another test.  Reduced the spacing between shots to ~75cm which seems to work well for a working distance of around 1m.  It's nice to see how much area you can cover quickly. Attached image is a medium quality dense cloud from just 15 photos (yes, I'm impatient  ;))  This building also has some dark areas so I tried using aperture priority for exposure.

Watched the shutter speed as I walked and took smaller steps if necessary to keep the exposure differences between consecutive photos to be around 1/3 stop so that was a better chance of getting good tie points between light and dark areas.
Title: Re: Taking fisheye to the extreme
Post by: bigben on January 28, 2015, 07:22:57 AM
Here's a cropped model of the building I started this testing with. https://skfb.ly/CJpR (https://skfb.ly/CJpR) This demonstrates some of the good points a bit better as well as some of the limitations and things I need to change.
Title: Re: Taking fisheye to the extreme
Post by: bigben on January 29, 2015, 10:30:10 AM
Finally got the point cloud up...
http://files.digitisation.unimelb.edu.au/potree/pointclouds/gatehouse_8mm.html (http://files.digitisation.unimelb.edu.au/potree/pointclouds/gatehouse_8mm.html)

This includes areas cropped out for the sketchfab model.  The next step will be a proper run with multiple lenses to try and put my new shooting strategy into practise. Considering the Polly Woodside as a challenge http://www.pollywoodside.com.au (http://www.pollywoodside.com.au)
Title: Re: Taking fisheye to the extreme
Post by: bigben on February 02, 2015, 10:50:20 PM
Reviewing the processing while I wait for some parts to finish my painter's pole rig.  Found an error in the masking of the gatehouse scene (my error) and am halfway through repeating the masks, although it shouldn't have that great an effect on the outcome. The one thing I had failed to consider though was the that there was still a lot of noise. I went back to the first test and ran though a few repeats of cycles of optimising and re-aligning and it made a pretty big difference.  https://skfb.ly/CLRQ (https://skfb.ly/CLRQ)

I pulled out the images of the alcove on the gatehouse as a quick calibration scene and repeated the process to get some better lens parameters and then re-ran the alignment on the entire project. Much better result with less noise and more points generated from the periphery of the images. I'll re-run it fully after I finish fixing the masks and I'll shoot my normal calibration scene today to refine the lens parameters...  Much happier with the prospects of using this lens now.
Title: Re: Taking fisheye to the extreme
Post by: bigben on February 06, 2015, 12:36:04 AM
After doing a rough calibration of my lens from some images in the previous post, I applied the settings to another project to test them out. The result was a pretty good improvement. Here's the pointcloud http://files.digitisation.unimelb.edu.au/potree/pointclouds/msd-atrium-optimised.html (http://files.digitisation.unimelb.edu.au/potree/pointclouds/msd-atrium-optimised.html)

The lower stairs and the floor below the atrium is a single camera path, 2 shots every 2-3 steps with camera pointing forward and back as I left the building. It's not a complete scan of the interior, just a test to see the results of different camera orientations in different situations.
Title: Re: Taking fisheye to the extreme
Post by: MeHoo on February 08, 2015, 02:58:52 AM
Your fisheye results are really making me want to invest in a wider lens than my 16-35mm.
Title: Re: Taking fisheye to the extreme
Post by: bigben on February 08, 2015, 08:39:39 AM
Your fisheye results are really making me want to invest in a wider lens than my 16-35mm.

The tests with circular fisheye images have certainly been very interesting but from a practical sense I think a full frame fisheye is a more practical lens in most situations. 

Longer term, I'm considering putting together to proposal for a local research grant to come up with a pole mounted rig that researchers could use for things like archaeological dig sites. I finally got my painter's pole set up with my DSLR and CamRanger which I'll use to figure out the camera/lens combinations and then hopefully get an internal grant to build it.
Title: Re: Taking fisheye to the extreme
Post by: bigben on February 08, 2015, 03:22:11 PM
Reprocessed my second test after doing a pseudo calibration of my lens and refining the masks. This made a HUGE difference. Alignment and dense cloud run at medium to save time. I even allowed myself a "wow" when I saw this.  ;)

https://skfb.ly/CPpn (https://skfb.ly/CPpn)

Updated the online pointcloud. Camera path did not go onto the road.
http://go.unimelb.edu.au/zj7n (http://go.unimelb.edu.au/zj7n)

Without using a specific reference scene it's very difficult to get good settings for the lens distortion of a circular fisheye, particularly the outer 20°.  The alcove of this building has so far proved to be the best one I've found... essentially a brick-lined box 2-3m wide would be good.

Canon 5D2, Sigma 8mm, 513 images
Title: Re: Taking fisheye to the extreme
Post by: igor73 on February 09, 2015, 01:40:05 AM
That is awesome!  I can see some uses for circular fish eye for sure! 

Title: Re: Taking fisheye to the extreme
Post by: igor73 on February 09, 2015, 01:52:12 AM


Could you please explain a bit more about the calibration process you mention using for this lens?  Do you mean you adjusted the mask so the black edges of the circular fish eye are removed?  Or do you mean you used Agisoft camera calibration tool?   
Title: Re: Taking fisheye to the extreme
Post by: bigben on February 09, 2015, 04:00:05 AM

Could you please explain a bit more about the calibration process you mention using for this lens?  Do you mean you adjusted the mask so the black edges of the circular fish eye are removed?  Or do you mean you used Agisoft camera calibration tool?

Yes, this is definitely something I have to write up.... but it goes something like this.

First, the lens calibration tool is not so practical in this case because of the extreme fov.

Secondly, you need to understand what is actually being calculated. The position of each point in an image is being "fixed" by recalculating it's distance from the optical centre of the image using an equation with the lens parameters as constants for the equation.  In order to do the reverse, you need to provide images where points in the scene appear in multiple images, in different positions spread from the edge to the centre of the image.  With a circular fisheye, this actually requires turning the camera, not just panning.

I prefer brick walls in reference scenes as you can visually assess whether the settings are good or not. Remember that this is a mathematical algorithm... Photoscan doesn't "see" bricks and straight edges.  The walls in the point cloud should be flat, with little noise, and the bricks should be rectangular and of consistent size.   If you look at a screengrab of the project I used to generate the lens parameters you can see the camera positions I chose to capture the confined space.  1. Walking back and forth through the space, and then a vertical pan with the camera looking across the space from opposite sides.  I probably would have done a grid of paths if I were shooting a calibration scene, but this seems to have worked in this case.

So then it's just a case of cycling through the alignment steps to refine the point cloud.
1. Align images
2. Gradual selection and removal of points using Reprojection error = 0.5 and Reconstruction uncertainty = 35 (it's important to do both in this case)
3. Optimise cameras (Do not optimise skew)
4. In the Camera calibration, save the Adjusted settings and load them into the Initial settings (this should chnage it to Auto to Precalibrated)

Repeat this a couple of times.  You know you're getting close when the maximum Reprojection error < 2.5 and Reconstruction uncertainty < 150

Once you've got your final settings I tend to Fix the lens parameters for the first alignment of a project to avoid the lens parameters getting re-calculated to values that will throw things out.  For testing, go out and shoot a big brick wall from a distance of 2m or less. If the bricks at the outer edges of the model are distorted then you may have some more work to do as this will have an adverse affect on calculating camera positions and create a lot more noise throughout the point cloud.

Full frame fisheyes are a bit easier to work with, and  probably wouldn't go below 18mp cameras for circular fisheyes.
Title: Re: Taking fisheye to the extreme
Post by: igor73 on February 09, 2015, 01:50:15 PM
I have a FX 16mm Nikon fisheye that i will try this with and a 36mpix Nikon d810.

Your calibration procedure is very interesting! This is new to me.  Thanks for sharing.   Would you calibrate your lens this way if you used a more normal non fish eye wide angle lens to? 
Title: Re: Taking fisheye to the extreme
Post by: bigben on February 09, 2015, 03:40:33 PM
I use a similar approach for all of my lenses, but the requirements are a bit more relaxed for narrower fovs.  A brick wall and cobblestone laneway worked well for standard lenses, and there's  a nice little corner of brick next to my window at work.  I've been trying other reference scenes for my circular fisheye on my camera (previous one was my work camera) and in the end I went back to same spot as my previous one.   I'll post some images later to demonstrate what incorrect settings look like. Much easier to explain with pictures.
Title: Re: Taking fisheye to the extreme
Post by: igor73 on February 09, 2015, 06:07:30 PM
And  you find that you get significant better alignment accuracy when using this method even on a normal wide angle lens?  I have to try this so please do keep the info coming.  Much appreciated! 

I have a very special application taking pictures underwater and the lens is behind a doom port that cancels out the water magnification effect but introduces its own distortion especially at the edge of the doom.  .  Do you think this could work using the tiles of a swimming pool as calibration target? 

Using Agisoft lens calibration is not practical as i can't take pictures of the monitor screen under water.  Also i was under the impression calibration was not really needed if you had EXIF data at least not for on fish eye lenses.   


Title: Re: Taking fisheye to the extreme
Post by: bigben on February 09, 2015, 10:46:31 PM
I wouldn't say it's a better method... just an alternative one.  I started using this for panoramic stitching software which uses similar parameters. (an old page on an unmaintained archive: http://archive.bigben.id.au/tutorials/360/calibrate/calibrate1.html (http://archive.bigben.id.au/tutorials/360/calibrate/calibrate1.html))

A swimming pool was the first thing that popped into my mind for your underwater application. ;)

EXIF data doesn't contain any info on lens distortion, so the distortion parameters always have to be calculated. If you don't use a calibration process to determine these beforehand, then Photoscan calculates them during the image alignment step. In this case, the accuracy of these values depends in part on the type of subject matter in the images and the choice of camera positions and orientations.   Other software like Adobe Camera RAW just use the lens make/model to look up a table of distortion parameters. For Photoscan though, you really need to do this for every camera/lens combination ... although you can use the values for k1,k2,k3 as starting points between cameras.

A statistical approach can also be useful... list all of the lens parameters for "successful" projects and compare them, looking for clusters of values that have the same pattern of being positive/negative e.g. for my fisheye lens k1,k2,k3 are -,+,-. In this case, though I'd just use these as starting values for k1,k2,k3 and let Photoscan optimise them.

Once you do have calibrated values, you can load them into projects and hit the "Fixed" check box in the Lens calibration window.  This will preventing incorrect values being calculated in some situations (especially important with circular fisheye lenses, less relevant to normal lenses)
Title: Re: Taking fisheye to the extreme
Post by: bigben on March 07, 2015, 10:26:50 AM
Finally learned to take my own advice and not optimise calibrated values... especially for fisheye lenses.  "Calibrated" my circular fisheye using grid of 6 camera stations, with each camera station using 3 rows at 0, 30 and 60° tilt, and 15° rotation increments.

Shot a complex object on my way home (building within a building).
Optimised lens parameters: https://skfb.ly/Dq7Q (https://skfb.ly/Dq7Q)
Fixed, calibrated lens parameters: https://skfb.ly/Dr6P (https://skfb.ly/Dr6P)
Title: Re: Taking fisheye to the extreme
Post by: bigben on March 10, 2015, 12:30:18 PM
The last few tests I did had more than enough overlap.  It was time to stretch things out a bit.  Another hour's shooting time, 310 circular fisheye images, camera on a 3.6m pole, slightly random camera path.  Camera was tilted downwards a bit too much, but a level camera and a more methodical camera path it would make a great set of images to complement a UAV survey, capturing the ground and lower building details that UAVs have difficulty with.
Medium quality alignment and dense cloud:  http://files.digitisation.unimelb.edu.au/potree/pointclouds/uom-poledancing.html (http://files.digitisation.unimelb.edu.au/potree/pointclouds/uom-poledancing.html)
Title: Re: Taking fisheye to the extreme
Post by: thesystemera on March 14, 2015, 09:47:09 AM
Fisheye and video, wow! Really impressive results guys. This is what I'm doing for my lazy sunday!
Title: Re: Taking fisheye to the extreme
Post by: erik leeman on March 14, 2015, 11:53:12 PM
Oh how nice it would be if the standard edition had this camera type also  :'(
I would gladly pay extra for this feature!
Title: Re: Taking fisheye to the extreme
Post by: mrb on April 15, 2015, 01:06:44 AM
Just picked up the Sigma 8mm f/2.8 for my walk-around Sony NEX-5T.  Really happy with the results, particularly interior scenes, after calibrating using Agisoft Lens.