1.6.4 ERROR_DEVICE_LOST

[BjFaeTorphins]

I can't get the "Build Texture" stage to run.

Tail of log...

2020-08-24 21:07:36 Camera contribution estimated (in 39.16 sec)
2020-08-24 21:07:36 Avg camera fetch time: 5.77944e-06 sec
2020-08-24 21:07:36 Avg camera processing time: 0.00271025 sec
2020-08-24 21:07:38 estimating cameras resolution...
2020-08-24 21:07:42 Cannot use GPU. Reason: VK Error : VkResult is "ERROR_DEVICE_LOST" at line 245
2020-08-24 21:07:42 Performing blending on CPU...
2020-08-24 21:07:45 calculating mesh connectivity...

v1.6.4

Error is repeatable.

[Alexey Pasumansky]

Hello BjFaeTorphins,

I see your support ticket in our helwpdesk system. It would be helpful, if you can share the mesh model that we can use for testing on our side in order to reproduce the GPU processing failure on a similar hardware configuration. So far we have not faced the problem on our test computers (for random dataset) but there are few similar reports.

[BjFaeTorphins]

Alexey

I provided the model to your support folks but have not heard how they got on.

Did they manage to replicate the problem?

Any idea if there will be a fix?

[Alexey Pasumansky]

Hello BjFaeTorphins,

There seems to be an internal bug related to the GPU based texturing for the high poly models (tens of millions of faces and more). We are planning to fix the problem in the next version update (would be definitely fixed in 1.7, but hopefully we will also port the bug fix to 1.6.5), but now as a workaround you can use the following tweak to disable GPU based texture blending: main/texture_enable_gpu - set it to False. Thus blending will be performed on CPU only.

[ilia]

I experience the same bug. And it is quite stable. Everytime I try mesh the same model it crushes. Should I upload it somewhere? If so: what is the best way to do it?

[Alexey Pasumansky]

Hello ilia,

Can you specify the OS version used, Metashape version and provide the processing log, related to the failed operation?

In case the following information is not included to the log, please provide it separately: GPU models used, GPU driver version installed.

[ilia]

OS: Win10
Metashape: 1.8.3 14331 64bit

Log:

2022-05-30 17:18:50 checking for missing images...Checking for missing images...
2022-05-30 17:18:50  done in 0.064 sec
2022-05-30 17:18:50 Finished processing in 0.064 sec (exit code 1)
2022-05-30 17:18:50 BuildTexture: mapping mode = Generic, page count = 18, texture size = 4096, texture type = Diffuse map, source data = Images, blending mode = Mosaic, fill holes = 1, ghosting filter = 1, enable_gpu = 1, relaxed_precision = 1
2022-05-30 17:18:50 Parameterizing texture atlas...
2022-05-30 17:18:50 Model size: 4829534 faces, 2419928 vertices
2022-05-30 17:18:52 Analyzing mesh...
2022-05-30 17:18:52 duplicate faces detected - fixed
2022-05-30 17:18:52 multiple edges detected - fixed
2022-05-30 17:18:52 total 70 faces removed
2022-05-30 17:18:52 Decimated to 999929
2022-05-30 17:18:52 Parameterizing texture atlas...
2022-05-30 17:18:52 Performing initial subdivision...   
2022-05-30 17:18:54 Division threshold 100000
2022-05-30 17:18:54 Got 224 initial parts.
2022-05-30 17:18:54 top 8 mean: 86110.8 stddev: 10606.1
2022-05-30 17:18:54 Done "chart_size_split" in 1.98
2022-05-30 17:18:55 ********************************************************************************************************************************************************************************************************************************
2022-05-30 17:19:45 Pack buffer size 1024 x 1024
2022-05-30 17:19:45 Scaling charts... 
2022-05-30 17:19:45 done. (in 0.084 sec)
2022-05-30 17:19:45 Concatenating charts... 
2022-05-30 17:19:45 done. (in 0.022 sec)
2022-05-30 17:19:45 Packing... 
2022-05-30 17:19:45 Parametrization contains 1226 charts
2022-05-30 17:19:45 Chart scaling stats min/5%/median/95%/max: 0.680041, 1, median=1, 1, 1
2022-05-30 17:19:45 Pack iteration 0 ratio 0.525
2022-05-30 17:19:47 Pack iteration 1 ratio 0.6625
2022-05-30 17:19:49 Pack iteration 2 ratio 0.59375
2022-05-30 17:19:56 Pack iteration 3 ratio 0.628125
2022-05-30 17:19:58 Pack iteration 4 ratio 0.610937
2022-05-30 17:20:06 done. (in 21.152 sec)
2022-05-30 17:20:06 Retrieving mesh
2022-05-30 17:20:23 Parameterizing done in 93.565 sec
2022-05-30 17:20:23 Blending textures...
2022-05-30 17:20:23 Found 1 GPUs in 0 sec (CUDA: 0 sec, OpenCL: 0 sec)
2022-05-30 17:20:24 Using device: NVIDIA GeForce RTX 3060, 28 compute units, free memory: 11120/12287 MB, compute capability 8.6
2022-05-30 17:20:24   driver/runtime CUDA: 11060/10010
2022-05-30 17:20:24   max work group size 1024
2022-05-30 17:20:24   max work item sizes [1024, 1024, 64]
2022-05-30 17:20:24 Found device NVIDIA GeForce RTX 3060 driver version 512.59.0.0, API version: 1.3.194, location 12::0
2022-05-30 17:20:24 Done "load_model" in 0
2022-05-30 17:20:24 All cameras are fine
2022-05-30 17:20:24 Done "feasibility_check" in 0.014
2022-05-30 17:20:24 Activating context...
2022-05-30 17:20:24 Basis camera type is uint8:3
2022-05-30 17:20:24 calculating mesh connectivity... done in 1.005 sec
2022-05-30 17:20:25 rendering 0 to 18 pages
2022-05-30 17:20:25 Initialized texture renderer
2022-05-30 17:20:25 Configuring pipeline...
2022-05-30 17:20:25 Initialized memory broker
2022-05-30 17:20:25 Configuring mosaic pipeline with outliers filtering
2022-05-30 17:20:25 Prefetcher image dims 6464 4860 3 1
2022-05-30 17:20:25 Constructed pipeline
2022-05-30 17:20:25 Relaxed precision enabled
2022-05-30 17:20:25 Collecting memory requests...
2022-05-30 17:20:25 Allocating memory requests...
2022-05-30 17:20:25 Estimated required video memory: 8148 MB
2022-05-30 17:20:25 Estimated device memory: total 12142 MB, used 375 MB, available 11374 MB
2022-05-30 17:20:25 Allocated 8148 MB of VRAM
2022-05-30 17:20:25 Allocated memory requests
2022-05-30 17:20:25 Configured pipeline
2022-05-30 17:20:25 Processing cameras...
2022-05-30 17:20:26 Loading mesh...
2022-05-30 17:20:26 Loaded 4829534 faces and 2474423 vertices.
2022-05-30 17:20:26 viewport estimation started
2022-05-30 17:20:26 ?**?*?*??*?*??*?*?*?*?*??*?*??*?**?*?*??*?*??*?*??*?*??*?*?**?****?***??*?*?***??***??****?****?*?************?*?**?*?**?*?**?*?**?*?**?*?**?*?*??*??*?**??*??***??*??*******??***??***??*??***??***??****?*??****?**?****?**?****?**?*?**?**?*?**?***??***??***??****?*?**?*??***??***??***??***??**??***??*?*?**??***??***??**??***?**??***?**??***?****?****?*****??***??***??***??*?*??*?*??*?*??*?*??*?*??*?*??***??***********?******************??***??****?****?****?*??****?*??****?*??****?*??****?*?*?***?*?*?***?*?*?***?*?*?***?*?*?***?*?*?*?*?*?**?*?**?*?**??*?*?**??*?*??*??*?*??*??*?*??*??*?*??*??*?*??*??*?****??*?****??*?****??*?**?*??*?**?*??*?**?*??*?**?*??*?**?*??*?**?*??*?**?*??*?**?*??*?**?*??*?**?*??*?****??*?*****?*?*****?*?*****?*?*****?*?*?***?*?*?***?***?*?*?***?*?*?***?*?*?****?**?*?**?*?**?*?**?*??*?**?*??*?**?*??*?**?*??*?**?*??*?**?*??*?**?*??*?**?*?**?*??*??*?*??*?**?*?**?*?*??**?*?*??**?*?*??**?*?*??**?*??*?*??*?*??*?*??*?*??***?***************?***********??*?*?*?*?******?*?*?*?*?*?*?*?*?****??***??*??*?*??*??*?*??*??*?*??*?*??*?*??*?*??*?*??***??***??***??***??***??***??***?****?****?****?****?*******?****?****?****??*?****?**?****?**?****?*??****?*??**?*?**?***??**?***??**?***??**?****?**?****?**?****?**?****?**?****?**?****?**??***?**??*?*??*??*?*??*??***??***?*?*?*?*?*?*?*?*?*??*?*??*?*?*?*?*?*?*?*?*??*?
2022-05-30 17:20:29 viewport estimation done. (in 3.404 sec)
2022-05-30 17:20:29 Avg camera fetch time: 6.89338e-06 sec
2022-05-30 17:20:29 Total stalled time: 0.007 sec
2022-05-30 17:20:29 Avg camera processing time: 0.00156101 sec
2022-05-30 17:20:29 resolution disparity estimation started
2022-05-30 17:20:29 ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
2022-05-30 17:20:44 resolution disparity estimation done. (in 15.032 sec)
2022-05-30 17:20:44 Avg camera fetch time: 1.76101e-05 sec
2022-05-30 17:20:44 Total stalled time: 0.004 sec
2022-05-30 17:20:44 Avg camera processing time: 0.018873 sec
2022-05-30 17:20:44 Average downscale stats min/5%/median/95%/max: 1, 1, median=1, 1, 1
2022-05-30 17:20:44 outliers detection started
2022-05-30 17:20:44 ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
2022-05-30 17:21:51 outliers detection done. (in 66.879 sec)
2022-05-30 17:21:51 Avg camera fetch time: 0.344962 sec
2022-05-30 17:21:51 Total stalled time: 3.768 sec
2022-05-30 17:21:51 Avg camera processing time: 0.0766755 sec
2022-05-30 17:22:00 Compressed texture pages. (in 8.57 sec)
2022-05-30 17:22:08 Compressed texture pages. (in 8.164 sec)
2022-05-30 17:22:08 texture detail/outlier estimation started
2022-05-30 17:22:08 *******************************Cannot use GPU. Reason: VK Error : VkResult is "ERROR_DEVICE_LOST" at line 46
2022-05-30 17:22:17 Performing blending on CPU...
2022-05-30 17:22:17 rendering 18 pages

[ilia]

I have some guess when and why it happens, but I'm really not sure abut it.
It happened 3 times on the same model on different part of it when I splitted it by smaller meshes to get best textures, but to fit these textures computation on GPU.

These 3 "sub-tiles" where it crushed had some complex structures where I can easily imagine some polygons got overlapped by other polygons several times from some points of view, had structures topologically close to torus or something more complex with genus higher than that. Not sure if it the case though.

The problematic "sub-tile" model has approx. 4m triangles. When I splitted the model even further or decreased the atlas from 4k x 20 to 4k x 10 the problem disappeared.

[Alexey Pasumansky]

Hello ilia,

The error  VkResult is "ERROR_DEVICE_LOST" usually indicates that there's an issue with the driver or hardware - the GPU wasn't able to handle the given instruction.

You can try to download the latest compatible NVIDIA driver (512.95 for the current moment) and make a clean driver install:
https://www.nvidia.com/Download/index.aspx

In case the issue with the texturing of the same model persists, please send to support@agisoft.com the following data from the project.files directory contents for the problematic chunk:
project.zip, chunk.zip, frame.zip and all the contents of /model/ sub-directory. We would then try to perform some tests with the synthetic imagery.


Some other tests that you can try:
- disable ghosting filter and try to rebuild the texture with the same parameters (it should reduce the VRAM requirements and GPU usage at certain step),
- try using 8K texture pages - 4-5 pages of this size should give the similar effective resolution of the texture.
Also check that there are no other applications running in parallel, which could use GPU intensively at some point (video rendering in the Internet browser, for example).

[ilia]

Dear Alexey,

Thank you for advice. I haven't encountered it again so far, but I keep running again a lot of reconstructions and if I encounter this again I will let you know. I switched to 8k textures. For sub-tiles of the main mesh it was better for me to have a smaller granularity of textures so I can calculated atlas size based on mesh area.