Slow alignment on new machine

[MikeM]

We're running into performance issues on a new Linux machine we just built. Alignment that is processing 2315 images is taking much longer to complete than on our old 2013 Mac Pro that only has one video card. We expected this new machine to be significantly faster at processing than the older hardware, so I think we're missing something here in regards to configuration. We have all 4 GPU's selected in preferences that show CUDA is enabled, the use CPU checkbox is unchecked. Output is being written to an iSCSI volume directly connected to machine. I'd appreciate any suggestions anyone may have on how we can get this machine to perform better. -Thanks
Machine specs:
AMD Epyc 7401p 24-core CPU x48 @ 3GHz
4 GeForce GTX 1080 Ti video cards
256 GB of RAM
500 GB NVMe SSD
OS: Ubuntu 18.04 LTS x64

[stihl]

I doubt photoscan scales properly with 24(!!) cores.. Even utilizing 12 cores seems to be a hassle sometimes..

So even though you have 24 cores, they're only running at 2 GHz with a turbo frequency of 3GHz (which is hard to maintain constantly). 2 GHz is rather slow even though there are 24 of them. If the software doesn't utilize the hardware properly, like in this instance, you'll end up having slower performance than a higher clocked i5 or i7!

Also having 4 graphic cards is also something that does not scale properly. You'll definitely not gain any linear performance boost by adding four. We at least did not see any increase in performance when adding a third GPU to our system.
The GPU's are also not used during the alignment step so you will only see any performance gain during the Build Depth Maps stage of the Build Dense Cloud step.

Is there a reason why you chose this particular set up? Is this system build specifically for Photoscan processing?

[Alexey Pasumansky]

Hello MikeM,

Have you checked the System Performance Monitor during the alignment stage (that took about 5 hours), whether all the CPU cores have been utilized?

Also when speaking about the comparison to another configuration, have you used the same processing parameters? Do you have the report from that run?

[MikeM]

I doubt photoscan scales properly with 24(!!) cores.. Even utilizing 12 cores seems to be a hassle sometimes..

So even though you have 24 cores, they're only running at 2 GHz with a turbo frequency of 3GHz (which is hard to maintain constantly). 2 GHz is rather slow even though there are 24 of them. If the software doesn't utilize the hardware properly, like in this instance, you'll end up having slower performance than a higher clocked i5 or i7!

Also having 4 graphic cards is also something that does not scale properly. You'll definitely not gain any linear performance boost by adding four. We at least did not see any increase in performance when adding a third GPU to our system.
The GPU's are also not used during the alignment step so you will only see any performance gain during the Build Depth Maps stage of the Build Dense Cloud step.

Is there a reason why you chose this particular set up? Is this system build specifically for Photoscan processing?

Thanks for your response, we had the same suspicion about the processor so you just confirmed it. This machine was specifically built for PhotoScan processing and the specs were based off of some PhotoScan benchmarks we found on pugetsystems.com.

[MikeM]

Hello MikeM,

Have you checked the System Performance Monitor during the alignment stage (that took about 5 hours), whether all the CPU cores have been utilized?

Also when speaking about the comparison to another configuration, have you used the same processing parameters? Do you have the report from that run?

Hi Alexey, we've checked system performance monitor and although you can see all cores being used for processing, it doesn't seem to be fully utilizing each core to it's full potential. The same parameters have been used as previous runs. The PDF shows the report from our old Mac Pro I referenced earlier. I can send you the full report or log if you'd like but they're far too large to be uploaded to the forum.

[Alexey Pasumansky]

Hello MikeM,

If you have the full processing logs, please sen them to support@agisoft.com. Also specify, if the camera type in the Camera Calibration dialog is set to Frame (default) or to Fisheye?

Also I see that the are thousands of markers in the project, do they have the projections on the source images and if so, whether the alignment has been started with those markers?

Maybe you can also share the project with the alignment results (just aligned cameras and tie point cloud), if it not very large, so that we can try to check the alignment time on our side basing on the same matching points.

[Alexey Pasumansky]

Hello MikeM,

Sorry for such long delay.

Currently basing on the project I assume that the following reasons are the main cause of the long alignment operation:
- the base frequency of AMD Epyc 7401p is 2 GHz, so since the alignment utilizes CPU only, the performance will be better on faster CPU. (from the report comparison it is possible to see that GPU-supported matching stage is about 3-4 times faster on your Linux machine compared to Mac Pro),
- big number of calibration groups due to the different image dimensions (result of cropping during scanning?),
- many false matching points seems to be detected by the edges of the images - according to the View Matches dialog each image have maybe a few hundreds matching photos due to the false information. If you are using scanned images, it may be reasonable to apply masks to the auxiliary information by the image borders, however, it could be much easier, if all the images have the same size, then you could just import the same mask image. Otherwise, you could mask the auxiliary information outside the actual image area only on the few images and then use Apply Masks to Tie Points option during image matching, so that all the tie points that could be found under the masks will be filtered out. This, however, shouldn't give huge impact, as I expect.