Can't Select Tile Size - Dual GPU V2.1.0.17526

[mrv2020]

Hi Alexey,

On the network, in the "Match photos" processing step, if the GPU mask "Use default GPU mask" and have Dual GPU, I get the message "Can't select tile size", I checked msg and it was there, GPU with memory zero.

Code: [Select]

2024-01-13 16:32:55 Free VRAM: 0 MB, maximum allocation required: 2047 MB, memory required: 0 MB
2024-01-13 16:32:55 [GPU] photo 5608: 40000 points
2024-01-13 16:32:55 Error: Can't select tile size
2024-01-13 16:32:55 processing failed in 9.154 sec

But the GPU drivers were updated.
 
Solution, I had to isolate the GPUs and defined Nmask "1,0" and "0,1" for each GPU and there was no more error message, completing the step without memory overflow.

Code: [Select]

2024-01-13 17:17:31 Found 2 GPUs in 0.182 sec (CUDA: 0.127 sec, OpenCL: 0.055 sec)
2024-01-13 17:17:32 Using device: NVIDIA GeForce RTX 2080, 48 compute units, free memory: 7110/8191 MB, compute capability 7.5
2024-01-13 17:17:32   driver/runtime CUDA: 12030/10010
2024-01-13 17:17:32   max work group size 1024
2024-01-13 17:17:32   max work item sizes [1024, 1024, 64]
2024-01-13 17:17:32   got device properties in 0 sec, free memory in 0.76 sec
2024-01-13 17:17:55 211487 matches found in 24.065 sec
2024-01-13 17:17:56 matches combined in 0.057 sec
2024-01-13 17:17:56 filtered 95567 out of 148497 matches (64.3562%) in 0.105 sec
2024-01-13 17:17:56 saved matches in 0.093 sec
2024-01-13 17:17:56 processing finished in 124.34 sec

Version 2.1.0 build 17526, in previous 2.0.3 same dataset without problems.

[Alexey Pasumansky]

Hello mrv2020,

How many GPUs are there available in the system? Can you share the GPU initialization lines for the problematic case, including the processing task parameters?

Also please specify the OS version and GPU driver version installed.

[mrv2020]

Hi Alexey,

Dual GPU RTX2080, Drive NVidia 546.33, Windows 10 Pro 22H2 19045.3930

Code: [Select]

2024-01-15 19:29:30 processing finished in 1.588 sec
2024-01-15 19:29:33 MatchPhotos.detect (1/100): downscale = 1, subtask = detect, generic_preselection = on, reference_preselection = on, filter_mask = on, mask_tiepoints = off, filter_stationary_points = off, keypoint_limit = 40000, keypoint_limit_per_mpx = 5000, tiepoint_limit = 4000, guided_matching = off, reset_matches = on, data_folder = F:/3762_01.files/1/0/point_cloud.1
2024-01-15 19:29:34 loaded keypoint partition in 0 sec
2024-01-15 19:29:34 loaded matching data in 0 sec
2024-01-15 19:29:34 Detecting points...
2024-01-15 19:29:34 Found 2 GPUs in 0.076 sec (CUDA: 0.043 sec, OpenCL: 0.033 sec)
2024-01-15 19:29:35 Using device: NVIDIA GeForce RTX 2080 SUPER, 48 compute units, free memory: 7110/8191 MB, compute capability 7.5
2024-01-15 19:29:35   driver/runtime CUDA: 12030/10010
2024-01-15 19:29:35   max work group size 1024
2024-01-15 19:29:35   max work item sizes [1024, 1024, 64]
2024-01-15 19:29:35 Using device: NVIDIA GeForce RTX 2080 SUPER, 48 compute units, free memory: 7110/8191 MB, compute capability 7.5
2024-01-15 19:29:35   driver/runtime CUDA: 12030/10010
2024-01-15 19:29:35   max work group size 1024
2024-01-15 19:29:35   max work item sizes [1024, 1024, 64]


[mrv2020]

Hello Alexey,

The new nvidia drive is available, try it on local and network with dual GPU, and the process stops with zero memory GPU.

 But if the local process and the dual GPU are turned on, the same thing happens, to solve it you need to turn off one GPU.

Another solution, alignment with medium accuracy, now the error occurs, takes more time, but can finish online, still a better option anyway, as it is faster than local.

Another point, when the error occurs on a GPU without memory, the gradual selection for projection accuracy is between 224-260 and reprojection over 1.15. The same dataset without errors with single GPU the value is now between 110-127 and error reprojection 0.6-0.8

[Alexey Pasumansky]

Hello mrv2020,

If you have time for additional test, please try to switch from CUDA to OpenCL implementation by setting up main/gpu_enable_cuda tweak to False value and then re-starting the same operation.

[mrv2020]

Hi Alexey,

With Cuda setup off, all GPU works fine, without issues.

The reprojection error 0.67 and projection accuracy 121.

[mrv2020]

Hello Alexey,


Nvidia published new driver 551.23, try with CUDA on and the issues continues.

[Alexey Pasumansky]

Hello mrv2020,

Just to confirm: do you observe the error when CUDA-processing is enabled and both cards are enabled regardless to the gradual selection filters applied?

Also can you confirm that depth maps generation doesn't give any errors in the same configuration (both GPUs are used and CUDA is enabled)?

[mrv2020]

Hello Alexey,

Yes, the error is presented when both are active and CUDA is enabled, regardless of the filter.

Yes, it presents errors in the depth maps when CUDA is active in the same configuration.

In time, I did a test with the alignment and activated max_workgroup_size=20 with CUDA, which caused processing to take longer and some errors, but no overflow occurred.

Another test was to align everything at downscale=(2) and the error did not occur either. But the depth map was successful limiting BuildDepthMaps/max_neighbors=10 and BuildPointCloud/max_neighbors=12, in both aspects I managed to reach 15, at most, and the error appeared again.

main/gpu_enable_cuda=false everything everything ran without errors but with high processing time and some isolated cases of errors in the depth map.