Forum

Author Topic: Slow pointcloud generation on nVidia Tesla K80 with Metashape v.1.5.0  (Read 10786 times)

adam_s

  • Newbie
  • *
  • Posts: 28
    • View Profile
Re: Slow pointcloud generation on nVidia Tesla K80 with Metashape v.1.5.0
« Reply #15 on: February 21, 2019, 02:24:43 PM »
Hello Alexey,

Meanwhile, i've updated Metashape to 1.5.1 version and i re-ran "Build dense cloud" process in the benchmark project.
Version 1.5.1 is slower than 1.5.0 in our case. The process took 26 min (depth maps) and 26 min (dense cloud) time. In previous version, the same process took 23 (depth maps) and 11 min (dense cloud).

I have done several tests to confirm the results.

Best regards,
Adam
« Last Edit: February 21, 2019, 02:27:23 PM by adam_s »

ruyi7952

  • Full Member
  • ***
  • Posts: 104
    • View Profile
    • My Blog
Re: Slow pointcloud generation on nVidia Tesla K80 with Metashape v.1.5.0
« Reply #16 on: February 24, 2019, 09:10:05 PM »
Hello Alexey,

Meanwhile, i've updated Metashape to 1.5.1 version and i re-ran "Build dense cloud" process in the benchmark project.
Version 1.5.1 is slower than 1.5.0 in our case. The process took 26 min (depth maps) and 26 min (dense cloud) time. In previous version, the same process took 23 (depth maps) and 11 min (dense cloud).

I have done several tests to confirm the results.

Best regards,
Adam

slow or fast it's Not standard.
Matching success rate and alignment accuracy are the essence.
If you want to be fast ot the fast, try version 1.3.0.
You'll find that the success rate of matching decreases.
Real Scene Model Professional Contractor

Collect、Process、Publish

adam_s

  • Newbie
  • *
  • Posts: 28
    • View Profile
Re: Slow pointcloud generation on nVidia Tesla K80 with Metashape v.1.5.0
« Reply #17 on: February 26, 2019, 05:03:09 PM »
Alexey,

Do you have any news in our case?

Best regards,
Adam

Mak11

  • Sr. Member
  • ****
  • Posts: 374
    • View Profile
Re: Slow pointcloud generation on nVidia Tesla K80 with Metashape v.1.5.0
« Reply #18 on: February 26, 2019, 08:06:20 PM »
Why is the OpenGL Vendor: Google Inc & not Nvidia for your Tesla K80s ?
Also the latest driver for your Windows Server 2016 config should be the 411.98  WHQL : https://www.nvidia.com/Download/driverResults.aspx/140697/en-us

Mak
« Last Edit: February 26, 2019, 08:10:25 PM by Mak11 »

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14813
    • View Profile
Re: Slow pointcloud generation on nVidia Tesla K80 with Metashape v.1.5.0
« Reply #19 on: February 26, 2019, 09:07:15 PM »
Why is the OpenGL Vendor: Google Inc & not Nvidia for your Tesla K80s ?

In case the application is started during Windows Remote Desktop session, it will be started in ANGLE compatibility mode.
Best regards,
Alexey Pasumansky,
Agisoft LLC

Mak11

  • Sr. Member
  • ****
  • Posts: 374
    • View Profile
Re: Slow pointcloud generation on nVidia Tesla K80 with Metashape v.1.5.0
« Reply #20 on: February 26, 2019, 09:44:50 PM »
Why is the OpenGL Vendor: Google Inc & not Nvidia for your Tesla K80s ?

In case the application is started during Windows Remote Desktop session, it will be started in ANGLE compatibility mode.

Oh, I didn't know that. Thanks Alexey.

Mak

adam_s

  • Newbie
  • *
  • Posts: 28
    • View Profile
Re: Slow pointcloud generation on nVidia Tesla K80 with Metashape v.1.5.0
« Reply #21 on: February 27, 2019, 04:35:13 PM »
Dear Alexey,
Dear Mak,

Do you think that "OpenGL Vendor: Google Inc" can be the source of our problem? Is it possible to force Metashape to launch in NON-ANGLE compatibility mode during Windows Remote Desktop session?

tutos

  • Newbie
  • *
  • Posts: 4
    • View Profile
Re: Slow pointcloud generation on nVidia Tesla K80 with Metashape v.1.5.0
« Reply #22 on: February 27, 2019, 11:43:43 PM »
I noted that the "generating dense cloud" step, after "dense cloud generation" is a ton slower in the last version. The CPU usage on 1.4.5 is almost 100% on my amd TR 2950x on all cores, and with 1.5.1 the CPU usage is very low. I hope this help.
« Last Edit: March 05, 2019, 11:12:59 PM by tutos »

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14813
    • View Profile
Re: Slow pointcloud generation on nVidia Tesla K80 with Metashape v.1.5.0
« Reply #23 on: March 05, 2019, 09:29:09 PM »
Hello Adam,

Currently we are trying to reproduce the considerable difference in processing time.

The difference in 20-25% that you are observing on benchmarking set, actually can be expected due to the changes in the depth maps generation procedure implemented in 1.5. These changes should result in more accurate depth maps.

But 200-300% slow down shouldn't be expected and it looks strange. We are trying to check, if it could be caused by dual-socket CPU configuration or double GPUs used. Were you able to observe the same slow down on the large project, after re-installing Metashape 1.5 (if system reboot is possible, it may be also considered)?
Best regards,
Alexey Pasumansky,
Agisoft LLC

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14813
    • View Profile
Re: Slow pointcloud generation on nVidia Tesla K80 with Metashape v.1.5.0
« Reply #24 on: March 07, 2019, 05:32:54 PM »
I noted that the "generating dense cloud" step, after "dense cloud generation" is a ton slower in the last version. The CPU usage on 1.4.5 is almost 100% on my amd TR 2950x on all cores, and with 1.5.1 the CPU usage is very low. I hope this help.
Hello tutos,

Would be helpful, if you could send the logs from both version runs and corresponding project report files (after generating the dense cloud) to support@agisoft.com.

Have you also checked the memory consumption during the operation? Probably, the disk swapping is used, which would explain low CPU activity and long processing time.
Best regards,
Alexey Pasumansky,
Agisoft LLC

TXPE

  • Jr. Member
  • **
  • Posts: 54
    • View Profile
Re: Slow pointcloud generation on nVidia Tesla K80 with Metashape v.1.5.0
« Reply #25 on: March 19, 2019, 07:45:53 PM »
I don't have quantifiable info right now, but I would also agree that Metashape 1.5 is considerably slower than Photoscan.  I'm running dual Xeons with 4 K80's (MS Azure VM).  My laptop with an i7-7700HQ and single 1080 can almmoooost keep up with the VM.  This dataset does have high overlap.  I've also tried the tweak.

Thanks,
Doug

TXPE

  • Jr. Member
  • **
  • Posts: 54
    • View Profile
Re: Slow pointcloud generation on nVidia Tesla K80 with Metashape v.1.5.0
« Reply #26 on: March 20, 2019, 04:46:31 AM »
I've attached a log from job just completed.  Correction, zipped log file exceeds allowable file size.
[Update] link to log file https://www27.zippyshare.com/v/VpipZP4C/file.html


1231 20MP photos
high overlap with about 120 oblique photos in the group
Quality: high
Filtering: Aggressive

2x Xeon E5-2690 v3
224GB RAM
4x Nvidia K80

Depth maps generation time 5 hours 53 minutes
Dense cloud generation time 3 hours 41 minutes

I have been connecting via Remote Desktop.
I tried connecting via RealVNC server and still see:
 OpenGL Vendor: Google Inc.
 OpenGL Renderer: ANGLE (Microsoft Basic Render Driver Direct3D9Ex vs_3_0 ps_3_0)
 OpenGL Version: OpenGL ES 2.0 (ANGLE 2.1.99f075dade7c)

« Last Edit: March 20, 2019, 06:44:51 AM by TXPE »

jansawicki

  • Newbie
  • *
  • Posts: 20
    • View Profile
Re: Slow pointcloud generation on nVidia Tesla K80 with Metashape v.1.5.0
« Reply #27 on: April 08, 2019, 03:42:45 PM »
Anything new? The issue still exists.

Greetings,

adam_s

  • Newbie
  • *
  • Posts: 28
    • View Profile
Dear Alexey,

I've performed some new tests and i've received very strange results.

My test project contains about 3000 photos (20 MP, aerial photos with typical photogrammetric overlaps).

When i disabled one of GPU, processing time was shorter than when Metashape used two GPUs. Difference is noticable - depth map generation step ~ 10h (one GPU) / 18h (both GPUs)  and point cloud generation step ~ 4h (one GPU) / 5.5h (both GPUs).

I attached below some printscreens presenting GPUs performence during depth map generation process. It is worth noticing that in case of both GPUs enabled, none of them reaches 100% of GPU load in opposite to case when only one GPU is enabled.

Are you working on some new update or optimization of GPUs supporting?

Best regards,
Adam

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14813
    • View Profile
Hello Adam,

Was "enable CPU" flag disabled in both cases and whether "fine-level task subdivision" option was enabled in the Advanced preferences for both runs?

Can you please additionally specify, if the images and project files are present on the local drive or are accessed via network?
Best regards,
Alexey Pasumansky,
Agisoft LLC