Forum

Author Topic: Alignment very slow on large project  (Read 7985 times)

dcobra

  • Jr. Member
  • **
  • Posts: 59
    • View Profile
Alignment very slow on large project
« on: December 16, 2024, 03:24:15 PM »
Hello,

I am processing very large projects with 60-70k images. I have successfully processed a few models of this size already using network mode and two nodes but there is a step in the alignment phase that is very slow. After detecting points, it comes to "x pairs selected in..." and then " setting point indices". This step takes days to complete and uses almost no resources on the computer. Is there a way to speed this up? The rest of the alignment processes as expected, using the resources available. I have tried using the --inprocess flag but it hasn't made a noticeable difference.

Thank you

Bzuco

  • Full Member
  • ***
  • Posts: 246
    • View Profile
Re: Alignment very slow on large project
« Reply #1 on: December 16, 2024, 04:42:13 PM »
Do you see at least GPU utilization or some network traffic between nodes? Selecting pairs and matching points are almost pure GPU tasks.

dcobra

  • Jr. Member
  • **
  • Posts: 59
    • View Profile
Re: Alignment very slow on large project
« Reply #2 on: December 16, 2024, 04:56:32 PM »
The only activity I see at this stage is a single core/thread near 100%. There is no GPU or network utilization. Before and after this step resources are well used.

dcobra

  • Jr. Member
  • **
  • Posts: 59
    • View Profile
Re: Alignment very slow on large project
« Reply #3 on: December 16, 2024, 04:58:40 PM »
Here is the console at this point:

MatchPhotos.select (1/1): downscale = 1, subtask = select, generic_preselection = on, reference_preselection = off, filter_mask = off, mask_tiepoints = off, filter_stationary_points = off, keypoint_limit = 60000, keypoint_limit_per_mpx = 1000, tiepoint_limit = 6000, guided_matching = off, exclude_corners = off, data_folder = //Be7960x/e/DVDSSP/PS/DVDSSP.files/0/0/point_cloud
loaded matching data in 0.01 sec
loaded matching partition in 24.507 sec
loaded matches in 280.017 sec
492448668 pairs selected in 19.8 sec

It will sit here for a long time and then move to "setting point indices", which also takes a very long time

Bzuco

  • Full Member
  • ***
  • Posts: 246
    • View Profile
Re: Alignment very slow on large project
« Reply #4 on: December 16, 2024, 05:28:54 PM »
In network scenario it seem to be a lot of work to join small partly results to one chunk, especially on large project.
Try to spawn more local workers on each node, it should speed up single threaded tasks.

dcobra

  • Jr. Member
  • **
  • Posts: 59
    • View Profile
Re: Alignment very slow on large project
« Reply #5 on: December 16, 2024, 05:32:44 PM »
Thanks for the suggestion. I am able to do this on some of the steps but unfortunately on this step there is only 1/1 worker. It doesn't seem to split the step into multiple parts.

Bzuco

  • Full Member
  • ***
  • Posts: 246
    • View Profile
Re: Alignment very slow on large project
« Reply #6 on: December 16, 2024, 05:38:18 PM »
Can you at least decrease keypoint limit to 40k or even 20-30k? Would it be sufficient for your project? Also tiepoints limit 6000...for my projects 2000-4000 was fully enough.

dcobra

  • Jr. Member
  • **
  • Posts: 59
    • View Profile
Re: Alignment very slow on large project
« Reply #7 on: December 16, 2024, 05:41:33 PM »
Thanks, I will try reducing and see if it has any affect on model quality for this project.

olihar

  • Sr. Member
  • ****
  • Posts: 282
    • View Profile
Re: Alignment very slow on large project
« Reply #8 on: December 16, 2024, 09:58:34 PM »
How fast is your CPU? This step you are talking about is driven by CPU. What are the specs?

dcobra

  • Jr. Member
  • **
  • Posts: 59
    • View Profile
Re: Alignment very slow on large project
« Reply #9 on: December 17, 2024, 02:05:30 AM »
Have tried on an i9 11900k and a threadripper 7960x. With both systems it hardly uses any CPU.

dcobra

  • Jr. Member
  • **
  • Posts: 59
    • View Profile
Re: Alignment very slow on large project
« Reply #10 on: December 17, 2024, 10:38:37 PM »
Can you at least decrease keypoint limit to 40k or even 20-30k? Would it be sufficient for your project? Also tiepoints limit 6000...for my projects 2000-4000 was fully enough.

Reducing the tie point limit did not reduce the number of pairs selected. I expect that reducing the key point limit won't either. The pairs selected likely has to do with the image pairs for matching, which has to do with the number of images and overlap rather than tie points or key points. If that is the case then I don't see any way to speed things up other than reducing the number of images.

dcobra

  • Jr. Member
  • **
  • Posts: 59
    • View Profile
Re: Alignment very slow on large project
« Reply #11 on: December 19, 2024, 04:47:49 PM »
In network scenario it seem to be a lot of work to join small partly results to one chunk, especially on large project.
Try to spawn more local workers on each node, it should speed up single threaded tasks.

I tried running this not in network mode and after getting to the same step it is still very slow and barely using any resources so it doesn't seem this is related to network processing.

Bzuco

  • Full Member
  • ***
  • Posts: 246
    • View Profile
Re: Alignment very slow on large project
« Reply #12 on: December 19, 2024, 11:06:01 PM »
And how does your project look like? It is drone photogrammetry of large area, or ground photogrammetry of some building, or...?

dcobra

  • Jr. Member
  • **
  • Posts: 59
    • View Profile
Re: Alignment very slow on large project
« Reply #13 on: December 19, 2024, 11:11:15 PM »
And how does your project look like? It is drone photogrammetry of large area, or ground photogrammetry of some building, or...?

It is underwater photogrammetry of a 3D object. The area is about 100m x 20m. Smaller projects that are ~20,000 images or less tend to get through that step relatively quickly.

Bzuco

  • Full Member
  • ***
  • Posts: 246
    • View Profile
Re: Alignment very slow on large project
« Reply #14 on: December 19, 2024, 11:31:47 PM »
Then the task which takes days has exponential dependence or at least non linear. You can try manually split the area on 2 or 4 regions and then join them in the end.
It seems to be the same behaviour like when copying many files in the same folder. When number of files are in thousands, everything is fast, but when it comes to houndreds of thousands of files, the copy process can almost stop, just several KB/s on any fast disk.