Forum

Author Topic: Using Spherical Panoramas /equirectangular images as an input for 3D reconstruct  (Read 12556 times)

ChristianeZhao

  • Newbie
  • *
  • Posts: 15
    • View Profile
    • Chistiane's LinkedIn
Original post:
https://www.linkedin.com/pulse/10-questions-related-agsioft-17-updates-using-spherical-zhao/?published=t


1. Does Agsioft support panorama images as an input to produce 3D point clouds?
Yes, of course! Agisoft Metashape/PhotoScan is the only professional 3D processing software that supports panorama as an input source. Surprised or not, Agisoft's software application was initially developed and modified based on the 360 images produced by the Civetta camera. As our founder and CEO at Weiss AG initially push forward such an idea to the Agsioft team years ago. In short, you can use spherical panoramas in equirectangular representation in Agisoft Metashape/PhotoScan.

Similar to any other photogrammetry method, 3D reconstruction is based on pattern recognition, so everything has to be visible for at least two different views; this does not change for the case of panorama images.

 

All you need is to load images, then go to the Tools Menu -> Camera Calibration window and switch the Camera Type from Frame (default) to Spherical for the calibration group that corresponds to the panoramic images.



2. New release in Agisoft Metashape 1.7, what does it mean to me?
Many users reported 1.7 has better depth map generation, out-of-core processing on GPUs, shorter process time, sufficient detail, and less noise in the resulting 3D model. However, it also means there is a greater demand for the image quality but lower hardware requirements.


Our technical teams tested and compared between two 3D point clouds from the same dataset but a different version of Agisoft Metashape. The result is that 1.7.0 shows a greater level of details in photogrammetry than any previous version. Yet as more objects are being recognized, we can also see a greater level of distraction on the point clouds, and it brings greater pressure if one wants to modify a dense point cloud. To deal with that, choosing an excellent automatic panorama HDR camera ensures a great quality of images and reduces the workload on dense point cloud modification.


3. I had been using Agisoft Metashape/PhotoScan for years, why I never know panorama images produce a better result than flat images in the 3D reconstruction?


Panorama images have been recognized and used as a source of input in 3D reconstruction in Agisoft's software. Yet, all the development take times, especially with the improvement of the panorama camera. With the continues development of Civetta in the past 12 years, the Civetta from Weiss AG remained as the highest precision HDR panorama camera in the market --- but not all user can afford it. It also means most panorama cameras available in the market cannot capture photos that are precise enough for processing and reconstruction of 3D point clouds in Agisoft Metashape/PhotoScan. With the accurate HDR panorama, it also means a pressure of processing hardware and longer office time. Still, with the new release of Agisoft Metashape 1.7, you can benefit from much quicker process time, while the quality also has significant improvement.



4. Does Civetta only provide panorama images as an input for 3D reconstruction in Agisoft Metashape/PhotoScan?

No, you can decide whether you would like to use HDR panorama images or flat images as an input. If you chose to use panorama as an input, you could also decide whether to use a JPG or EXR file. These can be set with Civetta Utility.




The advantages of working with pre-processed full spherical images are faster processing and better registration between the individual scan positions. It works well with a small project. Working with unprocessed photos directly from the Civetta provides a higher reconstruction density as it is directly retrieved from raw images.


The 32 bits EXR full spherical images should only be used when a 32-bit point cloud or 3D model is required. This is mainly for applications in the CGI/Film industry.





5. Why can Civetta reduce errors in Agisoft Metashape/PhotoScan processing?

Here are a few things that stitched panorama images cannot but the Civetta:

Reduce human error - Civetta only has one button, which means it can automatically generate an HDR panorama with only one click. It means the Civetta requires less training time and less human error ---- such as not enough overlapping space between images.
Less processing time - A measurement pair is two panorama images captured from a different height but on the same spot. For a measurement pair, Agisoft Metashape/PhotoScan only need to calculate the vector value of one image; then it can automatically know the other one.
Greater coverage but same lighting - Civetta has a vertical field of view as 180+ degrees and horizontal field of view as 360 degrees. As photos for the panorama images are automatically taken within 40 seconds, it minimizes the errors caused by lighting, moving objects and more.
A better result - If you still believe panorama images as an input is not your thing, you can always put all the HDR photos you taken from the Civetta to the Agisoft Metashape PhotoScan. Civetta and Civetta Utility can automatically adjust the lighting to make sure each photo pixel has an "average" lighting value. It enables better recognitions and matching in-between images. Our 3D point cloud results are stunning.
Measurable point cloud with great accuracy - Technically, the accuracy of a point cloud depends on many factors. On average, Civetta produces +-3mm accuracy with 3D reconstruction. 


6. If I am looking for high-quality panorama, what is the difference between using a panorama captured by Civetta and using a panorama stitched by Agisoft?

Panorama images are not well-known as an input source but an export source in Agisoft Metashape/PhotoScan. Only high precision panorama cameras can produce good point clouds, and they are not cheap. Agisoft Metashape/PhotoScan can stitch several images and create a panorama image.


So here is the question, if I can produce a panorama image via the Agisoft, why do we still need a Civetta? 

Civetta is not just a panorama camera but can automatically capture HDR images with 230 megapixels. With such high precision and wide dynamic range, it enables its user to adjust these images, especially using these precise, rich images as the raw input for entertainment, such as rendering and CGI.


Many companies including Apple, Shell, BP, Marvel and others are currently using Civetta to produce high-quality panorama source for their purposes. It is also proven that the Civetta can provide a lot more information in one single image, compared with a panorama that is stitched by Agisoft Metashape/PhotoScan or Photoshop.

When capturing 360°images on the location, Civetta produces data for 3D reconstruction and panorama images in 8 bit, 32 bit or as a measurement pair for photogrammetry applications.

Besides, its automatical image capture feature can reduce human error and processing time while improving results.



7. Why do panorama images from Civetta require less calculation time?
Because many of them are taken in the same position(with tripod), it ensures enough overlapping area between images. Working with well-calibrated 360° image matching is much easier due to the larger overlap area, producing less wrong matches.

Due to the high dynamic range (28 f-stops) and an optimized image processing workflow, est 3D reconstruction results are guaranteed.



8. Why is Civetta still the best panorama option when I can use a random camera to capture panorama?
This question has been answered above, but in short:

230 megapixels resolution, the most precise panorama solution you can find in the market for photogrammetry
Comprehensive coverage (360*180degrees)
Quick and easy to use (1 button & automatic capturing within 40 seconds)
Less calculation time
Less human errors/will not miss out
Suitable for all conditions - can be carry for air travel, waterproof


9. Do I need to purchase Agisoft Metashape/PhotoScan for panorama images if I purchased a Civetta already?
No, the Civetta Utility will come along with your purchase of Civetta. With the software, you will produce 230 megapixels HDR panorama images with several options available. It can be either JPG or EXR, 8 bits or 23 bits, depending on your needs. With precise and information-rich panorama images as input, your 3D reconstruction result on Agisoft will be high quality as well.



10. What is the general workflow of using panorama to produce 3D models in Agisoft?


Import images. You can choose to import a full set of spherical images, or ideally, measurement paired captured by the Civetta. It can be equirectangular (1:2 dimensions) full spherical images in JPG or EXR format based on your needs, but EXR format usually has a very high hardware requirement.
Camera calibration. Choose all the images and select "Spherical".
Masking. You should mask out the bottom area to reduce calculation time or error in Agisoft Metashape/Photoscan. Do not store your masked file as PNG but JPG.
Align photos. The aligning photo process enabled the machine to calculate the full spherical images' positions and connected one with another in the space.
Result. If there are outlying points in the point cloud (obviously wrong matches), you can mark them with the masking tools and delete them. Alternatively, you can use a gradual selection tool. With the slider, you can set the threshold for outliers. There should not be too heavy outliers, but still enough projection points left. The error (pix) should be in the 0.1+ area with more than 1000 projections. 


If you would like to add measurable value to your 3D model, you can mark a pair of images in the camera list and create a scale bar, with typing in 0.6 m in the distance field. Then you can run the optimize camera button, and you will get the calculated errors in between. This only work if you did a measurement pair in Civetta.

Measurement pair means you took two panorama images with Civetta in the same location, without moving the tripod.

Civetta can also use in combination with a laser scanner, a regular camera, drone and more; you just need to identify them in Agisoft Metashape/Photoscan before you process them.




About Christiane Zhao
Christiane is enabling a wider range of industries to adopt visual technology services to improve business performance. Christiane is currently working at the Weiss AG as a marketing officer. Weiss AG is a German High-Tech company with the legal form of a stock company. As a High-Tech company, our business objectives are the development, manufacturing and distribution of professional 3D and 360° capturing hardware and software for image processing and Visual Asset Management.



She is also working virtually for several other agencies and company. Christiane is always open as a freelancer.

https://www.linkedin.com/in/christiane-zhao-a073b4130/

Paulo

  • Hero Member
  • *****
  • Posts: 1302
    • View Profile
Hello Christiane,

it would be very interesting to have some Civetta 360 images to be able to process them with Metashape and test the quality of result. Where could we download some sample data set?
« Last Edit: January 16, 2021, 10:59:29 AM by Paulo »
Best Regards,
Paul Pelletier,
Surveyor

ChristianeZhao

  • Newbie
  • *
  • Posts: 15
    • View Profile
    • Chistiane's LinkedIn
Hello, Paulo.
Previously we did not provide dataset for download, but we received invitation from Agisoft team to upload some of the dataset and provide tutorials for those who are interested. At the meanwhile, you can check on our YouTube or LinkedIn channel, as well as our website to visualise the result.

YouTube: https://www.youtube.com/channel/UCXQvtIrbrpk6tdXgee7haoQ
LinkedIn: https://www.linkedin.com/company/weiss-ag
Website: www.weiss-ag.com

Please do not hesitate to contact us directly if you are interested. Feel free to find me in LinkedIn and schedule a meeting with me: ChristianeZhao

Thanks.

xabierr

  • Jr. Member
  • **
  • Posts: 84
    • View Profile
How much overlap/distance between 360 panoramas would you need for 3D reconstructions?

ChristianeZhao

  • Newbie
  • *
  • Posts: 15
    • View Profile
    • Chistiane's LinkedIn
Hello, xabierr.
Here is how Civetta works, with a click the camera will automatically turn 360 degrees can capture the scene, we recommend our users to capture measurement pairs( it means you will capture 2 panoramas with the exact same location but different height). The distance between each spot should be approximate 5-10 meters. In that case, it will automatically make sure it has enough overlaps.

The idea of using panorama in Agisoft is because it ensures a very high level of overlapping and it is a standardised process, so our users do not need to worry about how much overlap they will need to have to ensure the quality of the models. In addition:
1. It will be easier for the computer to recognise the common area between two different panoramas because they have a higher level of overlaps in between.
2. It means for measurement pairs, they only need to calculate one vector for two. So for 4 panorama images, Agisoft only needs to recognise 2 vectors. It is completely different from using your camera and captures thousands of photos, with thousands of vectors needed for the calculation.

In short, it ensures at least 60% overlapping.

Christiane Zhao
Feel free to follow me in LinkedIn: https://www.linkedin.com/in/christiane-zhao-a073b4130/

Kiesel

  • Sr. Member
  • ****
  • Posts: 332
    • View Profile
Hello Christiane Zhao,

that reads like a lot of advertising from your side and you might underestimate the existing knowledge about using panoramas in the Agisoft Metashape community.  Btw. there are existing some interesting papers about its use in Agisoft Photoscan/ Metashape already.

Would be nice if you upload some data so that we can play with it.

Best regards,

Kiesel
« Last Edit: January 22, 2021, 11:29:31 PM by Kiesel »

ChristianeZhao

  • Newbie
  • *
  • Posts: 15
    • View Profile
    • Chistiane's LinkedIn
Hello Christiane Zhao,

that reads like a lot of advertising from your side and you might underestimate the existing knowledge about using panoramas in the Agisoft Metashape community.  Btw. there are existing some interesting papers about its use in Agisoft Photoscan/ Metashape already.

Would be nice if you upload some data so that we can play with it.

Best regards,

Kiesel
Hello, Kiesel.
Thank you for your comments. On one side, I did witnesses people across the globe in different industries has benefited from using Civetta. You can check out some of the projects in the YouTube: https://www.youtube.com/channel/UCXQvtIrbrpk6tdXgee7haoQ

On other side, I am not having a photogrammetry and 3D backgrounds, so of course, there is a lot to learn and I am more than happy to hear advice from others.

We are invited by the Agisoft team to upload some of the raw data files that were captured by the Civetta, and it is in progress. We will let the community know once the Agisoft team completed the tutorial; or once these data are published for anyone in this community to use.

Feel free to leave some suggestions of what I should look at to make myself more qualify in this field.
Thank you Kiesel, and I am looking forward to hearing from you.
Cheers,
Christiane
https://www.linkedin.com/in/christiane-zhao-a073b4130/
« Last Edit: January 22, 2021, 11:31:44 PM by ChristianeZhao »

ChristianeZhao

  • Newbie
  • *
  • Posts: 15
    • View Profile
    • Chistiane's LinkedIn
Kiasel.
Sorry I miss out your question. For a single image, the size is ~ 21664 * 10832
You can check out here for sample images: https://share.weiss-ag.com/civetta230mp//#/-v/mainNodeId/0574e245-d03a-4527-8992-f5f61e0d2618 

Hello Christiane Zhao,

that reads like a lot of advertising from your side and you might underestimate the existing knowledge about using panoramas in the Agisoft Metashape community.  Btw. there are existing some interesting papers about its use in Agisoft Photoscan/ Metashape already.

Would be nice if you upload some data so that we can play with it.

Best regards,

Kiesel

Paulo

  • Hero Member
  • *****
  • Posts: 1302
    • View Profile
Christiane,

the image certainly very good resolution and crispness. But to play with in MS, we would need to get at least a stereo pair of 360 Civetta images....

Best Regards,
Paul Pelletier,
Surveyor

ChristianeZhao

  • Newbie
  • *
  • Posts: 15
    • View Profile
    • Chistiane's LinkedIn
Hello, Paulo.
Of course! As mentioned in previous post, we are working with the Agisoft team to upload these dataset for the community to use, they will also publish some tutorials about it.
Wait for the announcement!
Thanks.
Cheers,
Christiane Zhao
Christiane,

the image certainly very good resolution and crispness. But to play with in MS, we would need to get at least a stereo pair of 360 Civetta images....

bigben

  • Sr. Member
  • ****
  • Posts: 406
    • View Profile
...
You can check out here for sample images: https://share.weiss-ag.com/civetta230mp//#/-v/mainNodeId/0574e245-d03a-4527-8992-f5f61e0d2618 


First demo panorama has a number of stitching errors near the zenith.  Not something I'd expect for a system calibrated for photogrammetry.

Kiesel

  • Sr. Member
  • ****
  • Posts: 332
    • View Profile
Christiane,

thanks for the resolution, but found it in the meantime by myself. So for the Civetta we get a GSD of 1mm at a distance of 3.447m.

The stitching errors at zenith, found by Bigben, are really unexpected for a photogrammetric camera.

Btw. there are other cameras/devises with higher resolution like for example the Clauss Rodeometric (960Mpix) with additional functionality.

https://www.dr-clauss.de/en/vermessungstechnik-3/rodeonmetric

Regards,

Kiesel
« Last Edit: January 26, 2021, 10:36:19 AM by Kiesel »

ChristianeZhao

  • Newbie
  • *
  • Posts: 15
    • View Profile
    • Chistiane's LinkedIn
Hello, Bigen.
The weakest point of a panorama image is always the top area because it has less overlapping than the middle. Yes, these errors might create minor issues in image recognition, but it does not affect the quality of the model too much------ you will still get your 3D point cloud/ mesh from using these high-quality images as an input. Up to today, I do not think there is a product in the market that can capture 100% perfect panorama without any error. Even you use a Canon 5DSR to capture photos for 3D reconstruction, you might not use all the photos you captured right??  In addition, you can also put all the raw images taken into the Agisoft, that is another option. For Civetta, it is the most precise HDR panorama in the market, it means it has high precision and high dynamic range that can still create 3D reconstruction in over or underexposure environment. It can capture photos automatically, which means it requires less training and knowledge about photogrammetry.

If you still have doubt about the quality, feel free to check out these 3D models made by our clients in the YouTube: https://www.youtube.com/channel/UCXQvtIrbrpk6tdXgee7haoQ
You can also contact me in LinkedIn if you have further question or would like to schedule a meeting: Christiane Zhao.

Thank you.
Cheers,
Christiane

...
You can check out here for sample images: https://share.weiss-ag.com/civetta230mp//#/-v/mainNodeId/0574e245-d03a-4527-8992-f5f61e0d2618 


First demo panorama has a number of stitching errors near the zenith.  Not something I'd expect for a system calibrated for photogrammetry.

Shadow

  • Newbie
  • *
  • Posts: 24
    • View Profile
    • My 3D scans on Sketchfab
Very interesting thread !

As mentioned in previous post, we are working with the Agisoft team to upload these dataset for the community to use, they will also publish some tutorials about it.
Wait for the announcement!

Christiane, are there any updates on the dataset & tutorials ?

ChristianeZhao

  • Newbie
  • *
  • Posts: 15
    • View Profile
    • Chistiane's LinkedIn
Very interesting thread !

As mentioned in previous post, we are working with the Agisoft team to upload these dataset for the community to use, they will also publish some tutorials about it.
Wait for the announcement!

Christiane, are there any updates on the dataset & tutorials ?

Hello, you can check out my YouTube Channel, which includes some videos of using panorama in 3df and agisoft. You can search: Christiane Zhao.
For sharing dataset, we only share dataset to one with an interest in buying. In that case, I suggest you email us and schedule a zoom meeting with us.
Cheers,
Christiane