CUDA Crash during dense point cloud processing

[edtriplett]

Thanks for showing me this. I am still getting the same error in Standard. I attached the console text here.
I assume the DWORD item was 64bit, correct?

[edtriplett]

Sorry to bother again. I am posting this here because I can not add an attachment in a personal message.
We just purchased a Agisoft Pro license to work around this issue because this code worked the first time with the demo of Pro:

PhotoScan.app.settings.setValue("main/depth_max_gpu_multiplier", 1)

Unfortunately, I am getting the same crash again in Pro with this code pasted into the console before running the dense cloud process. I attached the console text here.

Again the error code is this:

Error: Kernel failed: the launch timed out and was terminated (6) at line 821

[Alexey Pasumansky]

Hello edtriplett,

To understand, if it is the hardware-related issue, I can suggest several tests:
- try to physically remove all GPUs but one and check, if the processing works or a similar issue occurs, you can also try putting GPU to another slot,
- if you have such an option - after the first test, try to put the GPUs (one by one) to another computer and also check, if the similar issue is observed on the second machine.

It may be also worth trying to run some CUDA-based stress tests on your configuration.

[edtriplett]

Hi Alexey,
I have gone back to the manufacturer of the PC and they suggested turning off a few CPUs in windows with the "Processor Affinity" option box. This did not have any effect. I am still getting the CUDA crashes.

I took your advice and detached all but one of the GPUS and everything worked fine. Attaching two or all three causes the crash. I am attaching the reports from Cuda-Z software showing the performance of each GPU. I hope that information is useful to you, because I can't interpret it.

[soulbank]

this happens to me all the time, i am on on a 4090t1 and i910k with 128gb ram on Windows 10 - when i finish processing one asset (in full so align / make mesh / texture / smooth / export model)  and then start a new project, after the initial alignment i get the same cuda error when i try to build the mesh

Note that i am processing a lot of assets back to back so it a pain to reboot every time (that fixed it). After a lot of headache i found a simple solution - in the Windows Device Manager, simply go to Display Adapters and Disable (wait a few seconds) then re-Enable your Graphics card.

The screen gets messed up (windows move around) but i am able to keep working without a reboot and its fast....

Seems that metashape is not cleaning up the CUDA que properly after finishing its jobs ? There is a way to reset the CUDA systems at the command line but i cant get it to work that way.