Alignment very slow on large project

[CheeseAndJamSandwich]

And how does your project look like? It is drone photogrammetry of large area, or ground photogrammetry of some building, or...?

It is underwater photogrammetry of a 3D object. The area is about 100m x 20m. Smaller projects that are ~20,000 images or less tend to get through that step relatively quickly.

I feel your pain.

My little uw scan of one of our dive sites took 5 days to process on my old Thinkpad.

What camera are you using?
I only had GoPro 7 cameras, but stuck 2 or 3 on a long pole (see my profile pic), to just get the scanning done 2-3 times faster.

What settings are you aligning with?
Using my gopro's uw images, it's just a fact that they're not as sharp as those from cameras on the top-side... We've got shit vis and dim light to deal with.
This meant that i don't see any benefit from using anything but Low for Aligning.  I use Low, 20,000 2,000.  This speeds up the run times insanely!  And i don't think it matters how many points there are, more the quality of them... And uw, we just can't compete with the drone photos!  I'd test Low 20,000 2,000 vs. your existing settings... To see if there's much difference. I'd guess not.

And again, as we're uw, we don't have any georeference data, no GPS data... So i think MS has to compare each photo with every other photo... This leads to exponential processing time increases as the number of photos increases...
I've got 40,000 images of my Manta Point dive site (see it in my signiture).... But to make it manageable, i landed up processing each dive individually, aligning it, then running the 'Reduce Overlap' command, set to 9.  This knocked out a third of the photos.  The nature of using gopros with 2 second timelapse intervals means i always have much more overlap going forward, as i can't swim fast enough! and gopros can't do 3 second intervals. damn.  Then i took each of the pared down dives, using the reduced number of photos, and then aligned them all together, from scratch. 4 days, lol.  And i then did another Reduce overlap.  And yes, i was learning as i was doing this, so my tracks around the dive site are all over the show, not wonderfully parallel 'mowing the lawn' tracks like metashape would want.  Only after that do i create the model... Which only took a day.
Aligning uw images without any georef data seems to be the hardest part for metashape.  The rest is easy.

I recently tried Bzuco's tip of running multiple local nodes.  This, for the alignment job, is much, much, much faster!!!  When i redid my big alignment, admitedly also with an aditional eGPU, but it reduced my 4 day alignment down to 1 day.
I'd say that you should definitely do what Bzuco suggested.  Start up multiple nodes on each physical machine.
On each of your 2 machines, run a subset of you data as a quicker benchmark.  Run it with one node, then 2, then 3, then 4, etc...  It seems to be the GPU that limits how many you can run.  I had a 1660 that worked the fastest with 8 nodes, but my newer 5700 XT only liked 6.  This is for the alignment job.  For building the model, i can only run 2.  So i just pause the ones i don't need.  Building the model chews up a lot of VRAM, so you must only run as many nodes as to keep it from maxing out the VRAM, and swapping to RAM, which spanners the processing speed.  When you're testing this, it takes a bit of time before it eventually gets to the VRAM dependant task, but if you see it swapping, just kill one of the extra nodes, and the server will just do that failed 'worker' at the end, completing the job just fine.  So what GPUs do you have???  How much VRAM do they have?
So, maybe you'll have 8 nodes on your i9, and 12 nodes on you Threadripper... So 20 nodes showing up in the Network Monitor.  And limit the i9 down to 2 after aligning, and the TR down to 3, perhaps...
Oh, and then i guess you'd want to pause the 'slower' computer's nodes when it's doing the single threaded stuff...  Now it's getting complicated!
Hopefully Metashape will get some AI stuff to know which node is fastest to push this single core stuff to... and to dynamically adjust the number of local nodes to suit each workstation's hardware...

So, try Low 20,000 2,000 and try multiple nodes per computer.

[dcobra]

Thanks for your thoughts. I do use multiple nodes on each computer, but the stage that takes forever can only be done by a single worker. The other steps in the alignment phase do benefit from the additional workers. The problem seems to be exponential with the number of images. Projects with ~20,000 images are fine and make it through this step fairly quickly and then move on to steps where multiple workers are beneficial. However, a project with ~70,000 images has been stuck in this step for almost a week and using hardly any resources.

[CheeseAndJamSandwich]

Sounds like it's stuck...
If you are using uw photos, then i'd very much recommend that you don't use High-60,000-6,000... UW photos just aren't that sharp.  And i've seen better alignment when using low, than with high!  Especially if some photos aren't aligning.

The alignment stage doesn't determine the 'quality' of the model... just the accuracy of the overall shape of it, it's just determining where each camera was.
Changing the alignment quality doesn't change how many faces get generated by Build Model...
To get higher quality models, i just turn up the 'Quality' in 'Build Model - General', It's Medium by default, but changing it to High, Ultra High, then you can see every polyp in 3D on the Staghorn Coral, even in 'Model - Shaded', without the texture!  Whereas on Medium, you just get bumpy branches.

Also, for what I've mostly been doing, i only really use Model - Shaded to produce images for display, printing.  The texture's view of a dive site map just looks soft and fuzzy, you see the bad vis again, but in shaded, you just see the shape of every coral, rock, sand ripple, etc. It looks like the site on an epic viz day! Much better for dive site maps, or survey area scans.  Processing the texture simply doesn't give me anything i want.  So i save a huge amount of processing time and energy by not even processing it!
BUT, for scanning coral frames, for our coral restoration project, then absolutely, the Texture makes it look exactly like a coral head, or coral frame. 

Again, what cameras are you using? What stuff are you scanning? And if you try Low-20,000-2,000, and produce the model, it might still give you what you're looking for. 
4 hours going to 40 hours is better than 10 hours going to 100!
Let us know if this does help.

[dcobra]

Processing isn't stuck. It progresses, just very slowly as it is only using one thread. Have tried reducing key points without any success. I'll try reducing tie points, which may help as it seems to be related to the number of matches. I thought this might have been the number of image matches but it may be tie point matches.

Sorry, I don't have many details on the camera used for the project other than it is ~4MP and a very wide field of view, ~180 deg.

[dcobra]

I tried reducing both key point and tie point limits and the step that takes a very long time is not affected by this which confirms that it is based on the number of image pairs rather than detected points. The other steps do process much faster though. Thanks for the suggestions.

[CheeseAndJamSandwich]

I tried reducing both key point and tie point limits and the step that takes a very long time is not affected by this which confirms that it is based on the number of image pairs rather than detected points. The other steps do process much faster though. Thanks for the suggestions.

Did you try running the alignment with the quality on 'Low'?  This will be what makes the difference.

When you don't have sharp images, as we do not underwater, then 'Low' sometimes works better, as it's looking for bigger features, rather than the sharp details.  And it'll just find an awful lot of backscatter, particulate matter when on high! Doh.
Low downscales the image using pixel binning.
But the 'Low' alignment doesn't affect the amount of details in the mesh, dense clouds later, it'll only affect the accuracy, the confidence in the positions of the cameras, that are later used to then generate the meshes and clouds, with whatever high/medium/low settings you generate them with.  What is the resulting data used for? Do you need cm/mm accuracy?  I'm guessing not.
I did a quick test on some of my scans, and Aligning low and high didn't change the number of faces in the resulting mesh.  But Building the Mesh with High, or Very High dramatically increased the number of faces, and the quality of the mesh.  It looked amazing... But obviously took ages longer.

I just wanted the whole scanned area to be accurate down to the rocks and corals, and not all distorted an weird scales... My resulting map came out excellent.  One day i'll reprocess it with GCPs, etc. to actually measure it.  But everything looks to be in the right place... Which is awesome, as it's a huge area!

Also, if you have data collected on multiple dives, you might wanna try what i did... align each dive separately, then use the Reduce Overlap command to thin out that huge number of photos...  Then you take those individual thinned out dives and combine them, and align them all from scratch, all together, to generate the full scanned area.  Reducing the overlaps helps a lot on the build mesh stages, etc. see the manual, help pages.

Try Low!