How to process a dataset with millions of photos ?

[JRM]

Hi everyone,

I would like to know if anybody here has implemented a process able to deal with millions of photos with Photoscan, if you have any feedback I am listening !

One of the dataset we are building contains +7 millions files and will double in the next months, all these pictures were taken with the same camera model and are georeferenced. The resolution is low (2008x1680, 3.37Mpx) but enough for our use cases.

While I am used to deal with massive lidar or raster datasets, I have not figured out how to achieve it with Agisoft Photoscan. Even if it is divided in 1400 alignment subsets which will themselves be subdivided with the split_in_chunck.py script, I don't know how to get to a seamless textured mesh without many manual interventions.

The chaintool I'm thinking of so far :
1. align 1400 subsets
2. divide them in chunk with an overlap for densification
3. export the point clouds and merge them back with PDAL
4. sample the cloud to get an homogeneous density (remove the overlap effect)
5. tile and mesh the output
6. bring ten of thousands of chunks back in PS for texturing

Problems :
1. python or manual work is required for loading cameras given a file list, is there a size limit for a project ?
2. python or manual work as it is not available as a batch process
6. python or manual work for loading the tiles and the referenced cameras, size limit of a project

And surely some others I can't think of right now

regards,
jr

[GPC]

Are you processing streetview imagery?

[JRM]

Yes, it is our biggest but the same issue arises for "smaller" ortho/oblique datasets with hundred of thousands photos at >36Mpx which requires a chunk-based processing.

[GPC]

I have to tell you, but you're using the wrong technology for this problem. I think a mobile laser scanner would solve your problem more easily.

[JRM]

This dataset was initially acquiered for a streetview use only, no 3D production planned as an aerial lidar campaign was done for it. However we now have one big stack of georeferenced photos with yaw/roll/pitch, a good overlap, a unique point of view... and a photogrammetry software, that's too tempting

[Alexey Pasumansky]

Our partners in Russia have scanned one of the country regions resulting in about 6 millions photos total, but the processing has been performed in blocks (about 50-60 thousands images per chunk for alignment, and each chunk could be split by 2-3 parts for the dense cloud generation).

I've seen a brief article related to that project recently, unfortunately, it's in Russian only:
https://sdelanounas.ru/blogs/110513/
The final model can be accessed via web-engine:
https://tomsk3da.admtomsk.ru/location/58?ll=56.469274641%2C84.947281716&z=659.93&heading=0.392&pitch=-0.833&roll=0.002&mode2D=false


But returning to your task, JRM, I have some doubts that any single-block detailed mesh can be generated for such large area. Maybe you can consider tiled model utilization instead.

[JRM]

Thanks for those links Alexey, I do not wish to get a single model in photoscan, the tiled processing is what I am seeking to achieve but the problem is how to get a seamless set of tiles with a minimal of handwork (all the file loadings, region and overlap defining, etc.).

The following screenshot is showing the border effect I would like to avoid