CL_OUT_OF_HOST_MEMORY | MS2.0.2 | Win11 22H2 | 2x RX 7900 XTX GPU | AMD 7950x

[andyroo]

I've been running incremental alignment on a new machine using dual AMD RX 7900 XTX GPUs and have incrementally aligned 20 photo sets so far running two at a time (two cameras on the aircraft). Matching just failed trying to align sets 21 and 22 and threw this error:

Error: ciErrNum: CL_OUT_OF_HOST_MEMORY (-6) at line 212

This is the first time I came across this error after doing incremental alignment on 20 other photo sets in the same project on this machine. I've been doing the same thing on a HPC and another workstation using NVidia CUDA GPUs for several weeks without encountering anything like this.  Log excerpt before the error is shown below, and the whole log for this run is attached as a zip. I'm going to reboot in case there's a GPU memory leak or something and will report back if it happens again.

Other details:

Driver Version
22.40.57.05-230523a-392837C-AMD-Software-Adrenalin-Edition

OS    Microsoft Windows 11 Pro N
Version   10.0.22621 Build 22621
Processor   AMD Ryzen 9 7950X
BaseBoard   ProArt X670E-CREATOR WIFI
Installed RAM   64.0 GB

Log excerpt:

Code: [Select]

2023-06-22 17:13:14 filtered 3261150 out of 3307520 matches (98.598%) in 0.448 sec
2023-06-22 17:13:16 saved matches in 0.007 sec
2023-06-22 17:13:18 loaded matching partition in 0.002 sec
2023-06-22 17:13:18 loaded keypoint partition in 0.001 sec
2023-06-22 17:13:46 loaded keypoints in 27.724 sec
2023-06-22 17:13:46 loaded matching data in 0.001 sec
2023-06-22 17:13:46 Matching points...
2023-06-22 17:13:48 AMD Radeon RX 7900 XTX (gfx1100): no SPIR support
2023-06-22 17:13:48 AMD Radeon(TM) Graphics (gfx1036): no SPIR support
2023-06-22 17:13:48 AMD Radeon RX 7900 XTX (gfx1100): no SPIR support
2023-06-22 17:13:48 Found 3 GPUs in 0 sec (CUDA: 0 sec, OpenCL: 0 sec)
2023-06-22 17:13:48 Using device: AMD Radeon RX 7900 XTX (gfx1100), 48 compute units, free memory: 24557/24560 MB, OpenCL 2.0
2023-06-22 17:13:48   driver version: 3516.0 (PAL,LC), platform version: OpenCL 2.1 AMD-APP (3516.0)
2023-06-22 17:13:48   max work group size 256
2023-06-22 17:13:48   max work item sizes [1024, 1024, 1024]
2023-06-22 17:13:48   max mem alloc size 20876 MB
2023-06-22 17:13:48   wavefront width 32
2023-06-22 17:13:48 Using device: AMD Radeon RX 7900 XTX (gfx1100), 48 compute units, free memory: 24557/24560 MB, OpenCL 2.0
2023-06-22 17:13:48   driver version: 3516.0 (PAL,LC), platform version: OpenCL 2.1 AMD-APP (3516.0)
2023-06-22 17:13:48   max work group size 256
2023-06-22 17:13:48   max work item sizes [1024, 1024, 1024]
2023-06-22 17:13:48   max mem alloc size 20876 MB
2023-06-22 17:13:48   wavefront width 32
2023-06-22 17:13:48 Loading kernels for AMD Radeon RX 7900 XTX (gfx1100)...
2023-06-22 17:13:48 Kernel loaded in 0.016 seconds
2023-06-22 17:13:49 Loading kernels for AMD Radeon RX 7900 XTX (gfx1100)...
2023-06-22 17:13:49 Kernel loaded in 0.017 seconds
2023-06-22 17:16:22 4156962 matches found in 154.515 sec
2023-06-22 17:16:23 matches combined in 0.35 sec
2023-06-22 17:16:23 filtered 3525814 out of 3579945 matches (98.4879%) in 0.487 sec
2023-06-22 17:16:25 saved matches in 0.006 sec
2023-06-22 17:16:27 loaded matching partition in 0.001 sec
2023-06-22 17:16:27 loaded keypoint partition in 0 sec
2023-06-22 17:16:56 loaded keypoints in 28.814 sec
2023-06-22 17:16:56 loaded matching data in 0 sec
2023-06-22 17:16:56 Matching points...
2023-06-22 17:16:59 AMD Radeon RX 7900 XTX (gfx1100): no SPIR support
2023-06-22 17:16:59 AMD Radeon(TM) Graphics (gfx1036): no SPIR support
2023-06-22 17:16:59 AMD Radeon RX 7900 XTX (gfx1100): no SPIR support
2023-06-22 17:16:59 Found 3 GPUs in 0 sec (CUDA: 0 sec, OpenCL: 0 sec)
2023-06-22 17:16:59 Using device: AMD Radeon RX 7900 XTX (gfx1100), 48 compute units, free memory: 24557/24560 MB, OpenCL 2.0
2023-06-22 17:16:59   driver version: 3516.0 (PAL,LC), platform version: OpenCL 2.1 AMD-APP (3516.0)
2023-06-22 17:16:59   max work group size 256
2023-06-22 17:16:59   max work item sizes [1024, 1024, 1024]
2023-06-22 17:16:59   max mem alloc size 20876 MB
2023-06-22 17:16:59   wavefront width 32
2023-06-22 17:16:59 Using device: AMD Radeon RX 7900 XTX (gfx1100), 48 compute units, free memory: 24557/24560 MB, OpenCL 2.0
2023-06-22 17:16:59   driver version: 3516.0 (PAL,LC), platform version: OpenCL 2.1 AMD-APP (3516.0)
2023-06-22 17:16:59   max work group size 256
2023-06-22 17:16:59   max work item sizes [1024, 1024, 1024]
2023-06-22 17:16:59   max mem alloc size 20876 MB
2023-06-22 17:16:59   wavefront width 32
2023-06-22 17:16:59 Loading kernels for AMD Radeon RX 7900 XTX (gfx1100)...
2023-06-22 17:16:59 Kernel loaded in 0.016 seconds
2023-06-22 17:17:04 loaded keypoint partition in 0 sec
2023-06-22 17:17:04 loaded matching partition in 0.033 sec
2023-06-22 17:17:05 loaded matching partition in 0.773 sec
2023-06-22 17:17:05 Error: ciErrNum: CL_OUT_OF_HOST_MEMORY (-6) at line 212
2023-06-22 17:17:05 Saving project...
2023-06-22 17:17:05 saved project in 0.111 sec
2023-06-22 17:17:05 Finished batch processing in 35851.5 sec (exit code 1)