Align taking exponentially longer w/ more tracks added...

[CheeseAndJamSandwich]

Is it normal that increasing the number of photos, exponentially increases the Align Photos time?
13,000 took 1 day
15,000 took 2.5 days!

What alignment settings may cause this?  I'm sure it's running with defaults.

I've started doing an underwater scanning project, mapping a rocky dive site.
It's a big site, so each dive I'm coming back with about 3,000 time-lapse photos from 2 GoPros (7+8) on a 3m pole (gives good data between them, and 2x quicker).
The tracks I'm swimming on each dive are obviously not nice and neat, mowing the lawn style tracks like a drone can give you... Instead there's a lot of overlapping, crossing, holes... And to add, the water visibility changes each dive, the colour of the water changes and the sun angle/clouds do too... There's also certainly a little too much overlaps going forwards i think... GoPros can't do 3s timelapses, only .5, 1, 2, 5, 10s, etc...  I use 2s...   So a nightmare for the Metashape i guess. Doh.
I have 7 pairs of camera tracks, which are in their own folders.  so 14 folders.

I've just aligned with the latest scans, starting with an blank model, adding the 7 pairs of tracks, totallying 15,000 photos... aligning on high.  Other settings default.....
On my Thinkpad P51 laptop, it took 2.5 days!!!  During the processing, i'm getting times like: "64302505 matches found in 1006.06 sec" during the "Selecting pairs..." cycles!
The previous time i ran it, with 13,000 photos, it was 1 day, and previously with fewer photos, much shorter.

After doing some gradual selection stuff, reduce overlap, it only takes a few hours to mesh and again to texture.  No need to create chunks.

I've attached some pics.
If you squint, you can see some manta rays i photoshopped into the capture from ms.  The mantas are 3-4m across.
And here's a decimated model in Sketchfab.
https://skfb.ly/oo7nq

I thought that the alignment was supposed to be the 'quick' part!
Help!
 

[CheeseAndJamSandwich]

EEEEEK!

I've added yesterday's scanning tracks... so another 3,000 photos...

Now i have a total of 18,000 cameras.

The previous processing run with 15,000 cameras was "Selecting Pairs", matches in 1000 seconds.
This processing run with 18,000 cameras is giving 1500 seconds!!!

So the alignment is looking like it's take much, much longer than the previous 2.5 days!!!

Please help!
Does anyone know how to prevent this exponential increase in matching time?

[c-r-o-n-o-s]

Did you start the alignment from the beginning each time?
Normally you only need to realign added photos.

[CheeseAndJamSandwich]

These last few runs were from scratch.

I have the "Keep key points" enabled, and have run it previously just by adding the new tracks the previous total alignment, but had a problem with one area where it didn't want to align, leaving an area with two layers of points...

My P51 is currently chewing thru the 18,000 photos, it's 'Selecting Pairs' still, and is up to 'matches31a.dat', so it might be a couple or three days at this rate.... So i can't test it for a while...


But generally, does adding extra missions/flights/tracks/dives worth of data to any data set increase the Align Time exponentially???

I really hope i'm missing something, as it's becoming unmanageable 

[c-r-o-n-o-s]

But generally, does adding extra missions/flights/tracks/dives worth of data to any data set increase the Align Time exponentially???

That always depends on the overlap and other factors.
But 18.000 pictures is also a VERY big project. (I think)
By the way, it is not always helpful to use the "highest" presettings.
If you don't need centimetre precision, I think the very low settings are sufficient for alignment.

[CheeseAndJamSandwich]

I've been trialling High, Medium and Low, but they don't seem to change the crazy long "Selecting Pairs" times...
I actually set the current run to low / 20,000 / 2,000...  I'm hoping it shortens the other steps of the align.

Yeah, i realise that this is quite a 'big' project...
Though i do think I've probably mapped 2/3, 3/4 of the area i 'need' to...  With further mapping being a plus...
For fun, here's a 4k animation of it so far:
https://youtu.be/BnSGENSdtmM

And obviously, that problem is exaggerated by my lowly ThinkPad P51 with its crappy old Quadro 2200M...

But i wasn't expecting the times to grow exponentially!

Alexey, can you shed any light on the Aligning process and the experiences i'm having?

Cheers.

[CheeseAndJamSandwich]

The lastest run took 3 days, 8 hours to align 18,000...  80 hours!!!

The mesh only took about 12, after a bit of gradual selection tasks and reduce overlap to 8.

Sadly, after creating the mesh, i've noticed that there's several areas where the's two 'layers' of mesh, where a surface hasn't aligned from different tracks...  This didn't happen on the previous run with 15,000, it was very well aligned... and i'm sure that the settings were the same (high, GP, 40,000, 4,000)... so i'm a bit frustrated by that.  Gonna have to redo it... 

The only setting that i can think of that may have changed, is "Adaptive camera model fitting"...  It's off for the latest run for sure...  Should this be on for my type of scans?

Cheers.

[MaciekK]

I'm sorry to say this, but do I understand correctly that you are doing a project of 18,000 photos on a laptop? Who are you expecting? Recently, I was making a project of 8,000 photos, the MS was about 12 hours (aligment, dense cloud, orto). I have ryzen7, 64gb ram, rtx2070 super, 1tb ssd samsung 970 evo plus. Just change the computer.

[wojtek]

It is actually quite impressive how fast this aligns considering what hardware you run it on 

[CheeseAndJamSandwich]

...Just change the computer.

I so, so wish i had some funds to do so!!! 

It was a decent spec workstation laptop 3 years ago... Damn you Mr Moore!

Send me any spare ones you have if you can? 

[CheeseAndJamSandwich]

The main point of this post was to inquire why the Align time gets exponentially longer.
I'm adding 20% more camers and it's taking 50% longer.

All other operations seem to scale linearly.  Meshing is fairly quick.

When it gets to "Selecting Pairs", and the next stage, it grinds to a halt, taking 1400 seconds per cycle.

Alexey, can you give us any insight into what MS actually does during the various stages of Align Cameras?

[Alexey Pasumansky]

Hello CheeseAndJamSandwich,

It would be helpful, if you could provide the processing logs related to each run (with different number of images), where you observe unexpected processing time.

[CheeseAndJamSandwich]

I have logging turned on now, for the redo i'm running, as the last run came back with lots of errors, double surfaces...  Even tho it was i think it was with the same default settings as the run previous version that gave an excellent result.

This time i'm running it low, 20k, 2k, instead of the normal high, 40k, 4k...

I've done a fair big of testing with different settings, but it doesn't seem to change much...  Basically, when it gets to Selecting Pairs, it takes the same enormous amount of time, no matter the settings.

When it competes, i'll attatch it.

But in the mean time, could you describe the processes, stages that MS goes through when it aligns?  And what things affect, help, hinder the align times?

[Alexey Pasumansky]

Hello CheeseAndJamSandwich,

It can be said, that Align Photos operation contains of two major steps:
1. Image Matching
2. Camera Alignment

Image matching stage starts with the feature points detection. The time spent on this task mainly depend on the Keypoint Limit parameter value and image dimensions. Then goes matching itself (may include the pre-selection sub-step). The time spent on this stage depends on the number of detected keypoints, number of images and pre-selection method used (also the excessive image overlap may slow down the process).
Camera alignment timing mainly depends on the number of images, number of calibration groups, number of detected tie points.

[CheeseAndJamSandwich]

I guess my erratic 'flight' (swim) paths are causing far too much overlap.  And stretching out processing time towards infinity...

So does the following sound like a reasonable plan?

If i start with flight 1, align it all, check it, and then do a reduce overlap.  Remove those disabled cameras, and save as "Flight 1's optimum cameras"
I've had problems with double skins when keeping points (set in prefs) and adding additional flights, so I'll delete the alignment.
Add flight 2's cameras and do a fresh align with 1's reduce cameras and all of 2's...  Then check, reduce, remove and save as "Flight 1+2's optimum cameras"
Repeat for every flight...

This will then only add one flight's worth of cameras that overlap too much, adding to an otherwise 'optimised' set of cameras.

I'd like to resume just adding flight to the kept points, so do you have any pointers to how to prevent double-skins?  I can't work it out.
And tips on the 'optimum' number to set when Reducing Overlap would be nice... There's no information on this value anywhere.