Forum

Author Topic: Best practices for handling 30,000 + images  (Read 16862 times)

feiko.lai

  • Newbie
  • *
  • Posts: 31
    • View Profile
Best practices for handling 30,000 + images
« on: February 24, 2017, 11:44:29 AM »
Hi all,
We are going start a aerial survey project which will have around 30,000 images collected. We will use Agisoft to process them and it is first time for us to handle such large amount of data.

So I want to ask for recommendation of both hardware and software sides:
- Recommended hardware configuration, ram, disk, GPU, CPU
- Cluster is necessary? If yes, how many nodes? The totally process time is reduced linearly to the number of nodes?
- Multiple chunks? How to balance number of chunks and the number of images per chunk.
- The best way to merge chunks? By markers? If yes, then we have to place GCP markers around the borders of flights?


Thank you in advance.

Thibaud Capra

  • Full Member
  • ***
  • Posts: 101
  • Master Student in Geodetic Engineering & Surveying
    • View Profile
    • INSA de Strasbourg, Topography Engineering (French)
Re: Best practices for handling 30,000 + images
« Reply #1 on: February 24, 2017, 12:16:38 PM »
According to Shervais, K. & Dietrich, J., 2016. Structure from Motion (SfM) Photogrammetry Data Exploration and Processing Manual. (Available at: http://kb.unavco.org/kb/article/structure-from-motion-sfm-agisoft-photoscan-processing-guide-848.html)

Quote
Processing time in Agisoft increases significantly with increasing numbers of photos. As such, it
is sometimes more efficient to split the photos into smaller portions, called chunks. The chunks
can be individually processed and then merged at any step, though they are generally merged
after the texture has been generated (i.e. the final step of processing).
However, the Merge Chunks operation does not work optimally and may delete portions of the
model or stitch the two chunks together inaccurately.
If absolutely necessary to use chunks because of the number of photographs and the processing
speed of your computer, the workflow below is the best to use:
• Align photos
• Enter GCPs
Duplicate the chunk multiple times (good rule of thumb: take the number of photos,
divide by 250, and then duplicate the chunk that number of times).
• Use the Resize Region tool to resize the bounding box in each duplicate chunk to 1/#
chunks in size. There should be some overlap between the edges of the bounding boxes to
help align the chunks later.

• Then process each chunk separately for the dense cloud and the mesh steps.
• Align the chunks. Merge the chunks.
• Export the DEM/Orthomosaic.

Bolded part is a good rule of thumb I'd say.

As for the clusters, I honestly don't know.
Best regards.
--
Thibaud CAPRA
Master Student in Geodetic Engineering, Cartography & Surveying
Master Thesis in Automated Processing of UAV-based Photogrammetric Data (ResearchGate Link)
INSA de Strasbourg, FRANCE
--

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14813
    • View Profile
Re: Best practices for handling 30,000 + images
« Reply #2 on: February 24, 2017, 02:49:56 PM »
Hello feiko.lai,

For better accuracy we usually suggest to align all the images at once, optimize them and then, if there are no sufficient resources, use duplicated chunks to build the dense cloud in the smaller region.

If you run the process in the network mode (even on one node) the memory consumption for some steps will be lower, also you can lower the memory consumption if you use lower key point and tie point limit, for example 20,000 / 4,000 for the parameters.

As for the issues that you have with the merging script, I can suggest to disable Merge Back function, so at least you'll get the chunks processed and could merge the data back manually, previously removing the unnecessary data and choosing only important data for merging.
Best regards,
Alexey Pasumansky,
Agisoft LLC

maddin

  • Full Member
  • ***
  • Posts: 161
    • View Profile
Re: Best practices for handling 30,000 + images
« Reply #3 on: February 24, 2017, 03:27:02 PM »
For better accuracy we usually suggest to align all the images at once, optimize them and then, if there are no sufficient resources, use duplicated chunks to build the dense cloud in the smaller region.

Those duplicated chunks - should they contain the same cameras, just some disabled? Or should all the cameras remain enabled but the reconstruction volume be reduced? Or should some cameras be deleted from some chunks?

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14813
    • View Profile
Re: Best practices for handling 30,000 + images
« Reply #4 on: February 24, 2017, 03:30:37 PM »
Hello maddin,

PhotoScan will use bounding box to select the cameras to be used in dense cloud generation process, so it doesn't matter much, if you leave all cameras as is, disable some distant cameras or remove them completely. However, if you are planning to merge the chunks back, you need to keep only one set of cameras in the chunks being merged to avoid duplicates of the same cameras.
Best regards,
Alexey Pasumansky,
Agisoft LLC

Vance

  • Newbie
  • *
  • Posts: 4
    • View Profile
Re: Best practices for handling 30,000 + images
« Reply #5 on: February 24, 2017, 08:49:50 PM »
Alexey gave me this piece of advice a few minutes ago, and it seems like it would apply to your project as well. I had a chunk that contained roughly 600 photos. I duplicated it 16 or so times to aid in processing. When I merged it back together, the resulting chunk had 10000 photos (with obviously around 16 duplicates of each photo, and became hard for my machine to handle.

Quote
Hello Vance,

The better way to get rid of the duplicate-cameras when you are duplicating the chunks is to group all cameras in every chunk (or in original chunk before duplicating) to the folder and then in the merged chunk you can just remove all but one folders with the duplicates.
« Last Edit: February 24, 2017, 08:52:23 PM by Vance »

tkwasnitschka

  • Jr. Member
  • **
  • Posts: 66
    • View Profile
Re: Best practices for handling 30,000 + images
« Reply #6 on: March 10, 2017, 08:24:53 PM »
I have a lot of issues with huge data sets as well:

1. For my fisheye camera, dense cloud reconstruction with a fisheye calibration fails to produce a coherent shape, but only in cluster mode... ?!?! My support ticket went nowhere whith this, I am afraid.

1. So I chose to manually split in chunks after alignment and merge back. What I found when I used the Split in Chunks skript is that the mesh borders where bounding boxes touch each other dont match up - at least in my data set which may be a bit noisy and dark. On the other hand, I dont want them to overlap as it will create a mess of intersections in the merged mesh.

3. My data set has 220k 20MP images. I divided it up into 25 overlapping chunks, which are then individually processed on a three machine cluster (64GB Ram each) with above issues. Although I froze the camera calibration, the overlapping camera poses do not perfectly match up when I align chunks to each other, meaning I have to distdrust the camera based alignment...

With all these issues, could it be that my camera calibration is just plain garbage? To me this is a plain but implausible explanation. I got the calibration from a sparse grid subsample reconstruction of 35k images (that did align in one go on bespoke 3 machines) which does not look distorted over the total 25 acres of coverage and matches an existing terrain model to within a meter.

photoscan_user

  • Jr. Member
  • **
  • Posts: 98
    • View Profile
Re: Best practices for handling 30,000 + images
« Reply #7 on: July 17, 2017, 12:11:52 PM »
@Thibaud Capra

What do you mean by ' Duplicate the chunk multiple times' ?

Should we do image matching and aligning step globally? and then do other steps on per chunk basis?
To build dense point cloud in chunk we need to load full global chunk (looks like also can hit RAM limit)?
And I wonder how large the RAM of machine should be to perform this global step for example for 30k photos?

Vikshut

  • Newbie
  • *
  • Posts: 3
    • View Profile
Re: Best practices for handling 30,000 + images
« Reply #8 on: April 09, 2018, 06:12:10 AM »
Hi,

I've got 30,000 aerial images that I need to process to get a single orthomosaic and DSM. I am getting a "Error: out of memory (2) at line 179." error when I try aligning all my images with reference preselection and medium accuracy.

My system config:
CPU: AMD Ryzen 7 1700X Eight-Core Processor 3.4 GHz
GPU: GEFORCE GTX 1070 Ti
Windows 10 Pro

The images are from a GoPro Hero 5, each with a size of about 2.2 MB.

Is there an alternative workflow I can use rather than having to align all the photos in the first step?
« Last Edit: April 09, 2018, 06:14:44 AM by Vikshut »

SAV

  • Hero Member
  • *****
  • Posts: 710
    • View Profile
Re: Best practices for handling 30,000 + images
« Reply #9 on: April 09, 2018, 10:00:49 AM »
Hi feiko.lai,

1) Follow Aleksey's advice  ;)
Quote
For better accuracy we usually suggest to align all the images at once, optimize them and then, if there are no sufficient resources, use duplicated chunks to build the dense cloud in the smaller region.

2. Use the 'split in chunks' Python script
https://github.com/agisoft-llc/photoscan-scripts/blob/master/src/split_in_chunks_dialog.py

3. Use a high-end machine (i7 10core processor, multiple GPUs, heaps of RAM) to process your imagery
Depending on your desired quality settings, you might not need to run it on a cluster.

All the best.

Regards,
SAV

 

Alain

  • Newbie
  • *
  • Posts: 17
    • View Profile
Re: Best practices for handling 30,000 + images
« Reply #10 on: May 11, 2018, 03:47:47 PM »
2. Use the 'split in chunks' Python script
https://github.com/agisoft-llc/photoscan-scripts/blob/master/src/split_in_chunks_dialog.py

How to run that Script ?
I just did run it unter "Tools->Run Script" (without any Arguments) but nothing happened.

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14813
    • View Profile
Re: Best practices for handling 30,000 + images
« Reply #11 on: May 11, 2018, 05:09:53 PM »
Hello Alain,

After using Run Script you should see a new element in the Custom Menu - run it to open the script dialog.
Best regards,
Alexey Pasumansky,
Agisoft LLC

Alain

  • Newbie
  • *
  • Posts: 17
    • View Profile
Re: Best practices for handling 30,000 + images
« Reply #12 on: May 15, 2018, 10:43:05 AM »
Ok, I see it now, it's not so obviuos ;-) Is it not possible to run the Dialogwindow after running the Script ?

I have 2000 Photos and wanna "Build Mesh" in High Quality, what settings do you recommend ? 2 x 2 ?
What means the Checkbox "Merge Back" ?

« Last Edit: May 15, 2018, 10:46:52 AM by Alain »