Boost computation speed for photo subalignment

[geo_enth3]

Dear Metashape team,

My task is to subalign images to a very big project. Speficially, I want to subalign ca. 100 images (on a daily basis) to a chunk which exists of ca. 27000 photos.

Every thing works fine and the photos get subaligned accurately. Here are the main parameters I am using:

downscale = 4
generic_preselection=True
keypoint_limit=10000
tiepoint_limit= 4000
keep_keypoints=True

However, it takes quite long to compute this (several hours) which is logical considering the project size. I still wanted to ask if you have any tips to reduce the computation times. Specifically could you:

a) recommend a hardware specifically for the subalignment task. My current setup consists of (CPU: AMD EPYC 7302 16-core; GPU GeForce GTX 1650; RAM: 206GB). I know this is not the ultimate hardware for this task but observing the task  manager during the process shows me that none of my resources are fully used so would a better CPU/GPU even make sense? If so, is there a specific hardware you would recommend for subalignment tasks?

b) How much (approximately) would the usage of externally measured camera coordinates (via a GNSS antenna mounted on the camera with an accuracy of a few cm) reduce the computation times (by activating reference preselection from source)?

Thank you very much in advance!

geo_enth

[jenkinsm]

I'm doing something similar and there is one setting that I ignored until recently, but setting this value has made my alignment process much faster and more accurate: Capture Distance. Try setting that and see if it speeds up the next alignment you attempt.

You should also be using Estimated preselection in addition to Generic preselection.

[Bzuco]

@geo_enth3
From HW perspective:  you are slightly loosing time in first phase(detecting points). Preparing data for GPU is single threaded task and If I am correct photos are decoded/resampled from JPEGs/... to uncompressed format in system RAM and from there data are sent to GPU. Your CPU has max. single thread clock 3.3GHz and according my tests detecting points speed is roughly linearly dependent. Latest desktop intel/amd CPUs has single thread clock attaking 5GHz and improoved internal caches, so preparing data for GPU is ~ linearly faster. Downside(maybe not) is support only 128GB(INTEL, did not check AMD) of system RAM. So for example: in detecting points phase you see very low GPU utilization.
As you mentioned, that you are adding just 100 images per day, then probably it is not worth invest into CPU which has high single threaded clock. But for completelly new project with thousands of images on day one it starts to make sense.

You can speedup image matching process if you enable use CPU when performing GPU processing in preferences. The speed increasment depends on how quick your cpu can prepare data for GPU and at the same time how many images can cpu cores process in comparisson with gpu.

Do not know if subalignment task needs selecting pairs and matching points tasks in such scale as in completely new project, but still, both tasks utilize heavily GPU, so you can only benefit...take graphics card which has higher raw GFLOPs performance.

[CheeseAndJamSandwich]

I'm doing something similar and there is one setting that I ignored until recently, but setting this value has made my alignment process much faster and more accurate: Capture Distance. Try setting that and see if it speeds up the next alignment you attempt.

You should also be using Estimated preselection in addition to Generic preselection.

Does Capture Distance help if there photos/camera don't have any geolocation info?  Like gopros, underwater?
If not, could it in the future?

[CheeseAndJamSandwich]

@geo_enth3
From HW perspective:  you are slightly loosing time in first phase(detecting points). Preparing data for GPU is single threaded task and If I am correct photos are decoded/resampled from JPEGs/... to uncompressed format in system RAM and from there data are sent to GPU. Your CPU has max. single thread clock 3.3GHz and according my tests detecting points speed is roughly linearly dependent. Latest desktop intel/amd CPUs has single thread clock attaking 5GHz and improoved internal caches, so preparing data for GPU is ~ linearly faster. Downside(maybe not) is support only 128GB(INTEL, did not check AMD) of system RAM. So for example: in detecting points phase you see very low GPU utilization.
As you mentioned, that you are adding just 100 images per day, then probably it is not worth invest into CPU which has high single threaded clock. But for completelly new project with thousands of images on day one it starts to make sense.

You can speedup image matching process if you enable use CPU when performing GPU processing in preferences. The speed increasment depends on how quick your cpu can prepare data for GPU and at the same time how many images can cpu cores process in comparisson with gpu.

Do not know if subalignment task needs selecting pairs and matching points tasks in such scale as in completely new project, but still, both tasks utilize heavily GPU, so you can only benefit...take graphics card which has higher raw GFLOPs performance.

If these tasks are single threaded...  are they doing/could they do concurrent decoding/resampling of multiple jpgs, to then feed the GPU?
If you have an 8C/16T cpu, could it spawn 8 or 16 decode/resample jobs, or perhaps only 4, or whatever, if it wasn't such a bottleneck?

If a task is stuck being single threaded, but working on discrete lumps of data (a photo), then spawning concurrent jobs seems to be obvious?
Possibly MS already does this...

My multi-day alignments could really do with some 'leaps' in processing performance to cut that time down... 

[Mak11]

Your GeForce GTX 1650 has to go. It is way to underpowered compared to the rest of the system. It was already a gimped low end SKU when released in 2019. Aim for an Nvidia RTX 3070 / AMD Radeon 6700XT.

Mak

[DayGeckoArt]

Your GeForce GTX 1650 has to go. It is way to underpowered compared to the rest of the system. It was already a gimped low end SKU when released in 2019. Aim for an Nvidia RTX 3070 / AMD Radeon 6700XT.

Mak

There are HP OEM RTX 3060 cards available now for under $500... Seems like the best bang for buck! They have over 4000 CUDA cores

[Bzuco]

If these tasks are single threaded...  are they doing/could they do concurrent decoding/resampling of multiple jpgs, to then feed the GPU?

I just launched two instances of MS, half of the images in each, and detecting points time dropped to 70%. So yes, feeding more frequently GPU with data would cut down time almost linearly untill one of GPU resources would reaches 100%.
But detecting points task is still just fraction of whole project processing time, so maybe from that reason preparing data for GPU is still single thread job.

[geo_enth3]

Dear all, thanks a lot for all your input. It will take some time to try and understand those suggestions myself so I will come back to you later but in the meantime I really appreciate your help!