GPU Turning off

[AnnieOwl]

Hello, my name is Andrey.

I am currently using the trial version of Agisoft Metashape Pro and planning to buy the license, but I have run into an issue.

I am using server\node architecture for processing my data. My server and node are running on a Linux machine without GUI that has an i5-9600k processor and 2 NVIDIA RTX 2080 Ti video cards.
Sometimes during the GPU processing steps (Match Photos, Build Depth Map) one of the video cards turns off (the same one every time) and both the server and the node freezes (i can communicate with the server via network monitor, but I can't stop the task or the node it's running on, and I can't kill the processes on the machine via terminal either).

I've tried to mask out the video card that turns off, but it did not help. It seems like my node uses both video cards no matter which mask I pass as an argument to my node.

[AnnieOwl]

I figured out that video card fans getting turned off when Agisoft computes GPU tasks (Match Photos, Build Depth Maps). That's why video card overheats and turns off. That happens with both video cards (I've tried launching the node with masks 1, 2 and 3).

Has anyone encountered such problems? Any ideas why this can be happening? I don't think my hardware is the problem because this same machine was used for machine learning before, and such errors never occurred. All the tasks are launched via python script if that matters.

Full characteristics of the machine:
CPU: Intel Core i5-9600K CPU @ 3.70GHz (desktop)
GPU: 2 x GeForce NVIDIA RTX 2080 TI Windforce 11G
RAM: 4 x DDR4 16Gb 2666MHz PC-21300 Kingston HyperX Fury
Motherboard: ASUS TUF Z390-PRO GAMING

OS: Ubuntu 18.04.2 LTS (GNU/Linux 5.0.0-37-generic x86_64)
NVIDIA driver version 418.56
CUDA Version 10.1.105

[Alexey Pasumansky]

Hello AnnieOwl,

Are you observing any similar on the same configuration if you start any CUDA/OpenCL intensive operation?

Metashape is not controlling GPU fans, so it might be some driver misbehavior.

[AnnieOwl]

This PC is fully loaded with machine learning tasks that use CUDA every weekend, and such errors never occur during this process.