Performance 1.2.5 vs 1.3.0 and gpu usage

[jiji]

Hello, admin

I was trying to test the performance between 1.2.5 and 1.3.0.  I have a dataset with 200 images, and I ran Alignement and buildDenseCloud.

I ran my tests on AWS EC2 instances with 4 different types of instances:

ec2Type    cpu  gpu      gpuType        ram( G)

g2.2xlarge   8   1   NVIDIA GRID K25     15   
g2.8xlarge   32   4   NVIDIA GRID K25     60

p2.xlarge   4   1   NVIDIA Tesla K80    61
p2.8xlarge   8   8   NVIDIA Tesla K80    488



I found that alignment is much faster on v1.3.0 than v1.2.5, which is very exciting !

But for the build dense cloud, I get some wired results, I found that the v1.3 is faster than v1.2.5 on g2.2xlarge and p2.xlarge (both with one GPU),  but much slower on p2.8xlarge (8 gpus) and g2.8xlarge (4 gpus).

I observed the gpu usage on these instances, I found that, for v1.3.0, only 10 - 15% of gpu is used on p2.8xlarge (8 gpus) and only 30 - 40 % gpu is used on g2.8xlarge (4 gpus).  But the gpu usage are higher on v1.2.5 on these 2 instances.

[Alexey Pasumansky]

Hello jiji,

There were some optimizations regarding GPU usage for the depth maps calculation since 1.3.0 build 3772. If you wish, I can send you a beta-version with those optimizations, so that you can perform additional tests. It would be also helpful so save the processing logs for the analysis.

[jiji]

Hello jiji,

There were some optimizations regarding GPU usage for the depth maps calculation since 1.3.0 build 3772. If you wish, I can send you a beta-version with those optimizations, so that you can perform additional tests. It would be also helpful so save the processing logs for the analysis.

I think I was using the 3772 build, 

"Agisoft PhotoScan Professional version 1.3.0 build 3772 (64 bit)" --> the message from the help of my photoscan

What do you mean that you have a beta-version ? the optimization is not in the build 3772 build yet ?

[Alexey Pasumansky]

Hello jij,

Yes, I mean that after release of the version 1.3.0 build 3772 some optimizations were implemented.

[jiji]

Hello jij,

Yes, I mean that after release of the version 1.3.0 build 3772 some optimizations were implemented.

Ok, cool, so I want that beta version for my tests,  if you can give me the download link, or you can send me by email : jieren172@gmail.com

thanks !

[jiji]

Hello jij,

Yes, I mean that after release of the version 1.3.0 build 3772 some optimizations were implemented.

Hello Alexey,

I just install your beta version my ec2 instance (linux), but i could not run it with a script option -r

I got this error :

QXcbConnection: Could not connect to display
photoscan-pro/photoscan.sh: line 19:  9370 Aborted                 "$dirname/$appname" "$@"

I did not have this error in the 3772 build.

Thanks

[Gaealice]

Hello jij,

Yes, I mean that after release of the version 1.3.0 build 3772 some optimizations were implemented.

Hello Alexey,

After installing your beta version, we rerun our tests with jiji .

We still observed that, only 10 - 15% of gpu is used on p2.8xlarge (8 gpus) and only 30 - 40 % gpu is used on g2.8xlarge (4 gpus). So it doesn't fix our issue.

[Jeremiah_ROWE]

Are your video driver settings at Maximum Performance power option?

[jiji]

Are your video driver settings at Maximum Performance power option?

I am not sure, but with the exact same gpu driver config, in our tests on v1.2.5, the gpu usage is much higer  on g2.8xlarge (about 60 - 70 %) and on p2.8xlarge (about 30 - 40 %)

[Alexey Pasumansky]

Hello Gaealice and jiji,

Can you provide more detailed information about the tests: number and resolution of the images used, processing setting, timing for each step (or full processing log)?

Comparison table of the processing time for both versions would be also helpful, if the GPU-supported tasks (depth maps calculation for 1.2) are separated from CPU-only tasks.

[jiji]

Hello Gaealice and jiji,

Can you provide more detailed information about the tests: number and resolution of the images used, processing setting, timing for each step (or full processing log)?

Comparison table of the processing time for both versions would be also helpful, if the GPU-supported tasks (depth maps calculation for 1.2) are separated from CPU-only tasks.

Hello Alex, i have sent the test results table and some logs to you (admin@agisoft.com), please check

Regards

[Alexey Pasumansky]

Hello jiji,

We have released version 1.3.1 that should behave properly for the systems with big number of GPUs.

[willfig]

I'm also experiencing much slower Dense cloud build times lately and in retrospect I think it started since updating to 1.3 and now 1.3.1. We run a network of four machines. For projects of 1000 photos it would take 2-4 hours to build the cloud on medium quality and moderate depth filtering. I've been running much smaller projects 400-500 photos with the same settings and it's taking sometimes 8-9 hours. We upgraded to 1.3.1 a few days ago and I think it's possibly a bit worse. I'm just responding here as I'm wondering if anybody else is experiencing similar issues.

[Alexey Pasumansky]

Hello willfig,

It would be informative, if you can provide the timing from the Chunk Info for the same project processed in 1.2.6 and in 1.3.1 with the same parameters.
Also if there's a full processing log for the Build Dense Cloud stage it would be also helpful.

Do you notice slow down on GPU-supported depth maps reconstruction step or during the depth maps filtering already that is performed on CPU only?

[PaleoPixels]

Same feeling here, not sure but awhile ago I did some tests in demo version 1.2.6 (CPU only) , dense cloud generation for 2000-3000 photo;s would take a day with the high quality preset. Now with 1.3.1 (not demo, normal license) it looks to be around 1,5-2 days (with dual GPU enabled).