match photos using only 25-50% of GPU capacity

[maurello]

I noticed that my nodes do not fully utilise GPUs during certain steps where GPU is engaged. For example Matching Photos during the alignment process does use GPU but only 25-50% of its capacity. CPU is also barely used, so it cannot be a bottleneck.

Is there an issue or something to set to fully utilise GPUs?

[Bzuco]

Hi, how many nodes are you using and what are the HW specs.? and alignment parameters?

Matching points process speed depends on Key point limit value...with lower values 5000-10000 GPU has very little amount of computation to do and after each batch of matching task some calculations are needed on CPU...and during that moment GPU is not utilized...therefor you can not see GPU utilization 100% all the time, but only 25-50 as you described in your case.
But if you increase Key point limit to e.g. 40000, then you will see, that GPU will spending more time @100%.

Key point limit doesn't make sense set to high value if you don't have large photos e.g. 24Mpix or higher. If you are using 4k photos from camera, then it is almost impossible to increase GPU utilization during matching points task. Maybe, if it is possible, devs could increase matching points batch size for GPU, but I don't see more in depth that process 

In other alignment processing subtask you can also see low GPU utilization, which can be caused by low disk read speeds, or by fact, that some process are CPU single threaded => GPU is not fed fast enough with data(first phase - detecting points).

[maurello]

Hi, how many nodes are you using and what are the HW specs.? and alignment parameters?

Matching points process speed depends on Key point limit value...with lower values 5000-10000 GPU has very little amount of computation to do and after each batch of matching task some calculations are needed on CPU...and during that moment GPU is not utilized...therefor you can not see GPU utilization 100% all the time, but only 25-50 as you described in your case.
But if you increase Key point limit to e.g. 40000, then you will see, that GPU will spending more time @100%.

Key point limit doesn't make sense set to high value if you don't have large photos e.g. 24Mpix or higher. If you are using 4k photos from camera, then it is almost impossible to increase GPU utilization during matching points task. Maybe, if it is possible, devs could increase matching points batch size for GPU, but I don't see more in depth that process 

In other alignment processing subtask you can also see low GPU utilization, which can be caused by low disk read speeds, or by fact, that some process are CPU single threaded => GPU is not fed fast enough with data(first phase - detecting points).

Clear, that is indeed my case. At the moment I am processing models extracting frames from 4K videos. I use the standard settings 40000 key points and 4000 tie points. In you message you mentioned that if I'd use 40000 as key point limit my GPU would run 100%, that's not the case. Maybe you meant tie point limit?

I do not see my disk drive nor CPU being a limiting factor at the moment. Disk reads/writes are far below the performance limit, so goes for the CPU. The overall feeling is that the processing idles at 25-50% capacity utilisation of my server resources, and I do not understand what causes the bottleneck.

[Bzuco]

I tested that 40000 number on my garden project and with 18Mpix photos(5184x3456 pix). For your underwater project ~8Mpix photos(~4k 3840x2160) different number will cause higher gpu usage, but!!! it also depends of the photo quality and gpu used. There for I was asking what is your HW specs. of your node.

For better explanation, I made 3 alignment test:
1. high accuracy(which is full photo resolution)
2. medium accuracy(which is half of photo resolution)
3. low accuracy(which is quarter of photo resolution)

As you can see from first screenshot, even if I set keypoint limit to 0(MS try to extract maximum number of feature points from photos), the maximum number key points was ~90000 for High, 15000 for Medium, ~4000 for Low quality....you can check console log in MS and see what are the point numbers in your underwater project for each gopro photo.

On second screenshot you can see that my rtx2060super gpu was in case High accuracy utilized ~16 seconds @100%. But in case Medium accuracy, there was not enough computation task in one batch for GPU and therefor MS was able to utilize GPU only for 0.8sec.@50-60%. It is even worse with Low accuracy... ...so there is no hardware bottleneck in your case... . Metashape(in its current version) is designed for larger tasks when using GPUs, therefor underwater project with 4k res. gopro photos cannot bring enough computational data for performant hardware.

Maybe there are some tweak parameters, which can increase batch size...so GPU can process more data at a time.

[maurello]

Clear, I did some tests with bigger size cameras taken via stills and it does indeed use more the GPU. Thanks!

[Alexey Pasumansky]

Hello maurello,

If you are observing low GPU utilization during the feature points detection sub-step, while during the image matching operation the GPU load is higher, then the reason of such behavior could be related to the time spent on loading the image from local/network disk and image decompression.
Due to these intermediate operations, GPU cannot be fully saturated while feature points are being detected, especially on low resolution images and on lower accuracy values, as Bzuco has already mentioned.