Forum

Author Topic: cudaMemGetInfo time out error  (Read 1413 times)

B_Free42

  • Newbie
  • *
  • Posts: 10
    • View Profile
cudaMemGetInfo time out error
« on: January 25, 2020, 05:29:51 AM »
When I first run Build Dense Point Cloud on Medium or Low setting I get this error after several seconds:

cudaMemGetInfo(&free_mem_size, &total_mem_size): the launch timed out and was terminated (6) at line 211

When I try running the same thing again, it errors out almost immediately and gives the same error message except that it terminated at line 33.

The Build Dense Point Cloud process will finish if I run it on "Lowest" quality setting.

Is this something I can fix? Do I need to change the amount of time the computer waits for the time out? Thanks!

I'm running Windows 10 Home on Dell G7 7790 laptop. GPU is NVidia GeForce RTX 2080 with Max-Q Design.

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 11878
    • View Profile
Re: cudaMemGetInfo time out error
« Reply #1 on: January 25, 2020, 12:12:13 PM »
Hello B_Free42,

Looks like the driver failure. After the first time out it has not recovered, so all the next tries fails immediately.

I suggest to make a clean driver install and check if it helps:
https://www.nvidia.com/Download/driverResults.aspx/156282/en-us

If the problem persists, please provide the complete processing log from the Console pane.
Best regards,
Alexey Pasumansky,
AgiSoft LLC

tpeachey

  • Newbie
  • *
  • Posts: 2
    • View Profile
Re: cudaMemGetInfo time out error
« Reply #2 on: February 07, 2020, 03:26:11 AM »
When I have my GeForce RTX 2070 selected under the GPU tab of Metashape preferences I have been having almost the identical issue when I run both 'Align photos' and 'Build mesh'. When I use only my Intel UHD graphics 630 selected, these processes will run successfully, albeit very slowly. I have reinstalled both the GeForce Game Ready Driver and the Studio Driver and restarted my computer before and after doing both.

This started out happening rarely and now happens every time and no process will run successfully. Here are 3 examples of error messages I've been getting;
"cudaMemGetInfo(&free_mem_size,&total_mem_size): unspecified launch failure (4) at line 211"
"cudaMemGetInfo(&free_mem_size,&total_mem_size): an illegal memory access was encountered (77) at line 40"
"Kernel failed: an illegal memory access was encountered (77) at line 235"

I have attached an example console file.

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 11878
    • View Profile
Re: cudaMemGetInfo time out error
« Reply #3 on: February 07, 2020, 02:26:39 PM »
Hello tpeachey,

Are you observing similar issues for other GPU-supported stages, like depth maps reconstruction, for example?

Does it help, if you re-install NVIDIA driver using Clean Driver Install option?
Best regards,
Alexey Pasumansky,
AgiSoft LLC

tpeachey

  • Newbie
  • *
  • Posts: 2
    • View Profile
Re: cudaMemGetInfo time out error
« Reply #4 on: February 09, 2020, 03:31:01 AM »
Thanks for your reply Alexey.

I have tried the 'clean' option when reinstalling with no success.

Are you referring to the depth reconstruction stage of the 'Build Mesh' process? Yes that seems to be when it is occurring in that process.

As an update, I did find that a 'Build Mesh' would complete on a previously aligned 133,000 point chunk at medium quality, though the resulting model was distorted and not representative of the object or the point cloud of the previously successful alignment.

I am including another console file which shows the above mentioned 'successful' Build Mesh and then following that a failed attempt to run a high quality Build Mesh on a 200,000+ point previously aligned chunk.

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 11878
    • View Profile
Re: cudaMemGetInfo time out error
« Reply #5 on: February 11, 2020, 09:06:47 PM »
Hello tpeachey,

It looks like a hardware problem, as the issues are observed during different processing stages. I can send you instructions, how to switch the GPU calculations from CUDA to OpenCL, but if the problem is related to the system, then most likely you'll get a kind of similar errors.

Meanwhile, you can try to run Memtest86 checks, just for case (both in multi- and single-threaded modes), just for case the problem is caused by RAM failures. If CPU, GPU or RAM are overclocked, it may be also a reason of the problem, however, it should be observed during other intensive calculation stages.
Best regards,
Alexey Pasumansky,
AgiSoft LLC