Forum

Author Topic: Tweak to limit GPU load during dense cloud generation  (Read 2168 times)

Pkroks

  • Newbie
  • *
  • Posts: 39
    • View Profile
Tweak to limit GPU load during dense cloud generation
« on: August 17, 2021, 04:26:26 PM »
Hi,

I am struggling to complete processing and seem to have an issue when I try to do processing on High or higher settings. Medium completes in no time..

I remember reading there is a tweak parameter that can be used in the preferences to limit the load placed on the GPU and CUDA cores to help the system to complete; it takes a bit longer but the load isn't that high.

I have previously had issues with this laptop of mine - I have reinstalled the latest Nvidia drivers from the nvidia website directly. I have also updated all windows and Agisoft and as of the last ten days or so I cannot complete any processing on High settings.

Any help is appreciated.

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 13211
    • View Profile
Re: Tweak to limit GPU load during dense cloud generation
« Reply #1 on: August 17, 2021, 09:07:06 PM »
Hello Pkroks,

Can you please describe the issues that you are observing (specify the GPU model used, OS version installed and GPU driver version) and provide the processing log from the Console pane, which   includes the error messages?
Best regards,
Alexey Pasumansky,
Agisoft LLC

Pkroks

  • Newbie
  • *
  • Posts: 39
    • View Profile
Re: Tweak to limit GPU load during dense cloud generation
« Reply #2 on: August 17, 2021, 10:27:38 PM »
Hi Alexey,

System specs -
Intel i7 8700k 3.70ghz 6cores
Nvidia GTX 1070 8gb ddr5
32gb RAM
samsung 850 evo 250gb ssd, and 1tb SSHD

Nvidia drivers version 471.68 installed off the nvidia downloads page. clean install after using Guru3d DDU uninstaller to clean previous versions.
Windows 10 profession 21H1 (build 19043.1165)

Error as shown in agisoft Logfile

2021-08-17 08:47:01 [GPU 1] Camera 110 tile #1/2: level #5/5 (x2 downscale: 1248x1824, image blowup: 2496x3648) done in 1.527 s = 33% propagation + 52% refinement + 8% filtering + 0% smoothing
2021-08-17 08:47:05 [GPU 1] Camera 110 samples after final filtering: 86% = 100% - 0% (not matched) - 5% (bad matched) - 0% (no neighbors) - 0% (no cost neighbors) - 7% (inconsistent normal) - 0% (estimated bad angle) - 0% (found bad angle) - 2% (speckles filtering)
2021-08-17 08:47:05 [GPU 1] Camera 110 tile #2/2: level #5/5 (x2 downscale: 1248x1824, image blowup: 2496x3648) done in 1.838 s = 32% propagation + 55% refinement + 7% filtering + 0% smoothing
2021-08-17 08:47:05 [GPU 1] group 1/1: estimating depth map for 25/49 camera 111 (30 neighbs)...
2021-08-17 08:47:08 [GPU 1] Camera 111 samples after final filtering: 70% = 100% - 0% (not matched) - 10% (bad matched) - 1% (no neighbors) - 2% (no cost neighbors) - 11% (inconsistent normal) - 0% (estimated bad angle) - 0% (found bad angle) - 5% (speckles filtering)
2021-08-17 08:47:08 [GPU 1] Camera 111 tile #1/2: level #5/5 (x2 downscale: 1248x1824, image blowup: 2496x3648) done in 1.519 s = 33% propagation + 52% refinement + 8% filtering + 0% smoothing
2021-08-17 08:47:12 [GPU 1] Camera 111 samples after final filtering: 83% = 100% - 0% (not matched) - 5% (bad matched) - 0% (no neighbors) - 0% (no cost neighbors) - 8% (inconsistent normal) - 0% (estimated bad angle) - 0% (found bad angle) - 2% (speckles filtering)
2021-08-17 08:47:12 [GPU 1] Camera 111 tile #2/2: level #5/5 (x2 downscale: 1248x1824, image blowup: 2496x3648) done in 1.812 s = 32% propagation + 54% refinement + 7% filtering + 0% smoothing
2021-08-17 08:47:12 [GPU 1] group 1/1: estimating depth map for 26/49 camera 112 (29 neighbs)...
2021-08-17 08:47:14 Error: cudaDestroySurfaceObject(cu_surface_): misaligned address (74) at line 126
2021-08-17 08:47:14 Warning: cudaStreamDestroy failed: misaligned address (74)
2021-08-17 08:47:15 Finished processing in 209.247 sec (exit code 0)
2021-08-17 08:47:15 Error: Kernel failed: misaligned address (74) at line 308


Pkroks

  • Newbie
  • *
  • Posts: 39
    • View Profile
Re: Tweak to limit GPU load during dense cloud generation
« Reply #3 on: August 18, 2021, 12:20:36 AM »
Now it won't even complete alignment without crashing..

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 13211
    • View Profile
Re: Tweak to limit GPU load during dense cloud generation
« Reply #4 on: August 18, 2021, 03:43:16 PM »
Hello Pkroks,,

Thank you for additional information.

We'll try to analyze the log and check, if there's anything wrong on Metashape side. Meanwhile we are trying to reproduce the issue on similar hardware configuration, but so far matching and depth maps generation works on GPU without any problems.
Do you have any CUDA-intensive applications that you can run on the same machine to check, if the problem could be related to the hardware problems with the GPU?

Also I can suggest to try switching to OpenCL mode by setting main/gpu_enable_cuda tweak to False value via advanced preferences tab. Can the same processing operation be finished without any errors? If there are still any GPU-related errors, please provide the related part of the log.
Best regards,
Alexey Pasumansky,
Agisoft LLC

Pkroks

  • Newbie
  • *
  • Posts: 39
    • View Profile
Re: Tweak to limit GPU load during dense cloud generation
« Reply #5 on: August 19, 2021, 12:01:13 AM »
Hi Alexey, I tried your tweek and switched it to OpenCL, it does the same and crashes the application.

All these problems only started when I installed the latest version of windows 10 pro with 21h1. I will attempt to roll this back.

I have also attempted using Reality capture to test and it is not working either, similar CUDA error. And unigene heaven tests are displaying artifacts and crashing. Hopefully rolling back to an earlier windows will help, otherwise it seems my GPU hardware may have freaked out..

will update you after I do a clean install with an older version of windows. It may be worth noting the errors also occur with both the Studio drivers and the Game ready drivers for Nvidia.

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 13211
    • View Profile
Re: Tweak to limit GPU load during dense cloud generation
« Reply #6 on: August 19, 2021, 08:24:26 PM »
Hello Pkroks,

Thank you for additional information.

We'll also try to use the 21H1 update for Windows 10 and see, if it causes any similar GPU-related processing problems.
Best regards,
Alexey Pasumansky,
Agisoft LLC

Pkroks

  • Newbie
  • *
  • Posts: 39
    • View Profile
Re: Tweak to limit GPU load during dense cloud generation
« Reply #7 on: August 30, 2021, 07:26:30 PM »
Hi Alexey, it seems it must be something wrong with my graphics card. I will replace the card and see if it fixes the problem.

Thanks

Pkroks

  • Newbie
  • *
  • Posts: 39
    • View Profile
Re: Tweak to limit GPU load during dense cloud generation
« Reply #8 on: September 22, 2021, 09:33:56 PM »
Hello Pkroks,

Thank you for additional information.

We'll also try to use the 21H1 update for Windows 10 and see, if it causes any similar GPU-related processing problems.

Hi Alexey, it seems that the laptop in question is actually over heating. The actual hardware is working but the cooling is inefficient and the temp seems to get too high and it throttles back limiting the card use to then prevent any permanent damage.

I just replaced the thermal paste again with new stuff, cleaned out the vents and it processed on high settings.

Is there a way to limit the load on the card so it doesn't get to such high temps, understandably it would take longer to do processing, but I would limit the load until I can find a permanent or more efficient way to cool the laptop sufficiently.

temps on the card were getting up to max of 87C and sitting around 80C during depth maps.

Once it gives the error I have to restart the laptop to then try processing again.

Bzuco

  • Jr. Member
  • **
  • Posts: 52
    • View Profile
Re: Tweak to limit GPU load during dense cloud generation
« Reply #9 on: September 23, 2021, 11:08:07 AM »
You can download MSI afterburner util and set power limit to what value you need, eg. 70%. The power limit range depends on GPU and how vendor set that in bios.
Lowering power limit will decrease automaticaly core clock, so you can increase it back using GPU core clock slider...in my case I am able with 70% power limit increase core clock max. +120MHz and it is stabel. Find your best value from +0MHz and increasing it with +20MHz steps. ...This is how you can run your GPU with lowest voltage at some frequency and also with much lower temperature.

If you have RTX card, then you can additionaly use nvidia driver utility with this command:  "C:\Program Files\NVIDIA Corporation\NVSMI>nvidia-smi.exe -i 0 -lgc 300,1000"
-i 0   is index of your GPU, first one.
-lgc  will set minimum and maximum core clock frequency, in my example min 300MHz and max 1000MHz
- with this util you have more control on your GPU
« Last Edit: September 23, 2021, 11:10:54 AM by Bzuco »