Refine Mesh: Kernel Failed: invalid argument (1) at line 99

[kb2zuz]

I am trying to refine a mesh. I ran it with the settings of Ultra High, 10 (also previously tried iterations, and 0.5 smoothness

The project has 126 aligned images but disabled 40 of them and am trying to only run on 86 images that were cross polarized and not the 40 that were not polarized. All images are 50MP (8272x6200 and 16 bit/channel TIFFS).

Here's a copy and paste from the console:

2023-11-08 09:56:09 Found 2 GPUs in 0.139 sec (CUDA: 0.083 sec, OpenCL: 0.056 sec)
2023-11-08 09:56:09 RefineModel: downscale = 1, iterations = 10, smoothness = 0.5
2023-11-08 09:56:09 Initializing...
2023-11-08 09:56:12 Found 2 GPUs in 0 sec (CUDA: 0 sec, OpenCL: 0 sec)
2023-11-08 09:56:18 Using device: NVIDIA GeForce RTX 4090, 128 compute units, free memory: 22599/24563 MB, compute capability 8.9
2023-11-08 09:56:18   driver/runtime CUDA: 12020/10010
2023-11-08 09:56:18   max work group size 1024
2023-11-08 09:56:18   max work item sizes [1024, 1024, 64]
2023-11-08 09:56:18   got device properties in 0 sec, free memory in 6.584 sec
2023-11-08 09:56:19 Using device: NVIDIA GeForce RTX 4090, 128 compute units, free memory: 22998/24563 MB, compute capability 8.9
2023-11-08 09:56:19   driver/runtime CUDA: 12020/10010
2023-11-08 09:56:19   max work group size 1024
2023-11-08 09:56:19   max work item sizes [1024, 1024, 64]
2023-11-08 09:56:19 Device 'NVIDIA GeForce RTX 4090' has 22209 Mb of free memory
2023-11-08 09:56:19 Device 'NVIDIA GeForce RTX 4090' has 22608 Mb of free memory
2023-11-08 09:56:19 Analyzing model...
2023-11-08 09:56:20 Faces: 3154562, Vertices: 1578983
2023-11-08 09:56:20 Memory required on each device: 2555 Mb + 661 Mb = 3217 Mb
2023-11-08 09:56:20 Using device 'NVIDIA GeForce RTX 4090' in concurrent. (2 times)
2023-11-08 09:56:20 Using device 'NVIDIA GeForce RTX 4090' in concurrent. (2 times)
2023-11-08 09:56:33 Stage #1 out of 2
2023-11-08 09:56:33 Faces: 3154562, Vertices: 1578983
2023-11-08 09:56:33 Memory required on each device: 638 Mb + 661 Mb = 1300 Mb
2023-11-08 09:56:33 Preprocessing model...
2023-11-08 09:56:40 Faces: 3669557, Vertices: 1836481
2023-11-08 09:56:40 Memory required on each device: 2807 Mb + 769 Mb = 3577 Mb
2023-11-08 09:56:41 Loading photos...
2023-11-08 09:57:02 Error: cudaFree(data_): invalid argument (1) at line 108
2023-11-08 09:57:02 Error: cudaFree(data_): invalid argument (1) at line 108
2023-11-08 09:57:02 Error: cudaFree(data_): invalid argument (1) at line 108
2023-11-08 09:57:02 Error: cudaFree(data_): invalid argument (1) at line 108
2023-11-08 09:57:02 loaded photos in 20.686 seconds
2023-11-08 09:57:02 Refining model...
2023-11-08 09:57:12 Iteration #1 out of 10
2023-11-08 09:57:30 Iteration #2 out of 10
2023-11-08 09:57:47 Iteration #3 out of 10
2023-11-08 09:58:04 Iteration #4 out of 10
2023-11-08 09:58:22 Iteration #5 out of 10
2023-11-08 09:58:39 Iteration #6 out of 10
2023-11-08 09:58:56 Iteration #7 out of 10
2023-11-08 09:59:13 Iteration #8 out of 10
2023-11-08 09:59:31 Iteration #9 out of 10
2023-11-08 09:59:48 Iteration #10 out of 10
2023-11-08 10:00:05 Stage #2 out of 2
2023-11-08 10:00:05 Faces: 3669557, Vertices: 1836481
2023-11-08 10:00:05 Memory required on each device: 2555 Mb + 769 Mb = 3325 Mb
2023-11-08 10:00:06 Preprocessing model...
2023-11-08 10:00:25 Faces: 10209967, Vertices: 5106849
2023-11-08 10:00:25 Memory required on each device: 11231 Mb + 2142 Mb = 13373 Mb
2023-11-08 10:00:25 Using device 'NVIDIA GeForce RTX 4090' in concurrent. (1 times)
2023-11-08 10:00:26 Using device 'NVIDIA GeForce RTX 4090' in concurrent. (1 times)
2023-11-08 10:00:26 Loading photos...
2023-11-08 10:00:27 Error: cudaFree(data_): invalid argument (1) at line 108
2023-11-08 10:00:27 Error: cudaFree(data_): invalid argument (1) at line 108
2023-11-08 10:00:27 Finished processing in 257.605 sec (exit code 0)
2023-11-08 10:00:27 Error: Kernel failed: invalid argument (1) at line 99

[Alexey Pasumansky]

Hello kb2zuz,

How many polygons are there in the original model (about 3 millions) and what is the NVIDIA driver version installed and OS used?

[kb2zuz]

Original Model: 3,154,562 faces

NVIDIA GeForce RTX 4090 Driver Date: 6/8/2023 Driver Version: 31.0.15.3623

WIndows 10 Pro, Version 22H2 OS Build 19045.3570

[Alexey Pasumansky]

Hello kb2zuz,


Looks like you are observing the similar problem as reported before:
https://www.agisoft.com/forum/index.php?topic=13591.0

The identical CUDA cards cause the problem for Refine Mesh operation. As a workaround in the current version you can switch to OpenCL implementation by setting main/gpu_enable_cuda tweak to False value.

[kb2zuz]

I can confirm that the work around was successful.