Forum

Author Topic: Hard system crash consistently during depth map filtering  (Read 4669 times)

Dannosuke

  • Newbie
  • *
  • Posts: 13
    • View Profile
Hard system crash consistently during depth map filtering
« on: August 01, 2019, 03:06:39 AM »
Hi everyone,

I am having a consistent problem. My computer locks up with a black screen during the depth map filtering process which requires a hard reset. I have no problem running any other program or bench marking program. But every time I try to create a dense point cloud the computer locks up.  It lets me create a sparse point cloud no problem. I'm running Metashape on a Threadripper 2990wx processor, with two Radon VII GPUs, and 128 GB of RAM.

thanks,

3DFranz

  • Newbie
  • *
  • Posts: 44
    • View Profile
Re: Hard system crash consistently during depth map filtering
« Reply #1 on: August 01, 2019, 11:23:39 AM »
I am also oberserving problems during DepthMaps Filtering. (DepthMaps Generation is working, but Filtering not!)
My System isn't freezing, but Agisoft Metashape terminates the Program, without an error Message.
This Program Termination isn't accuring all times. Sometime Metashapes is closing during Depth Filtering, sometimes it is not.
Any advise?

System:
Windows 10 Pro
Intel i9-9900 x
128 GB Ram
2 x Nvidia RTX 2070


2019-08-01 10:11:22 Agisoft Metashape Professional Version: 1.5.3 build 8469 (64 bit)
2019-08-01 10:11:22 Platform: Windows
2019-08-01 10:11:22 CPU: Intel(R) Core(TM) i9-9900X CPU @ 3.50GHz (desktop)
2019-08-01 10:11:22 CPU family: 6 model: 85 signature: 50654h
2019-08-01 10:11:22 RAM: 127.7 GB
2019-08-01 10:11:22 OpenGL Vendor: NVIDIA Corporation
2019-08-01 10:11:22 OpenGL Renderer: GeForce RTX 2070/PCIe/SSE2
2019-08-01 10:11:22 OpenGL Version: 4.6.0 NVIDIA 431.60
2019-08-01 10:11:22 Maximum Texture Size: 32768
2019-08-01 10:11:22 Quad Buffered Stereo: not enabled
2019-08-01 10:11:22 ARB_vertex_buffer_object: supported
2019-08-01 10:11:22 ARB_texture_non_power_of_two: supported

....
2019-08-01 10:17:05 checking for missing images...Checking for missing images...
2019-08-01 10:17:05  done in 0.097 sec
2019-08-01 10:17:05 Finished processing in 0.097 sec (exit code 1)
2019-08-01 10:17:05 BuildDenseCloud: quality = High, depth filtering = Aggressive, point colors = 1
2019-08-01 10:17:05 Generating depth maps...
2019-08-01 10:17:05 Initializing...
2019-08-01 10:17:05 Using device: GeForce RTX 2070, 36 compute units, free memory: 6736/8192 MB, compute capability 7.5
2019-08-01 10:17:05   driver/runtime CUDA: 10010/5050
2019-08-01 10:17:05   max work group size 1024
2019-08-01 10:17:05   max work item sizes [1024, 1024, 64]
2019-08-01 10:17:05 Using device: GeForce RTX 2070, 36 compute units, free memory: 6691/8192 MB, compute capability 7.5
2019-08-01 10:17:05   driver/runtime CUDA: 10010/5050
2019-08-01 10:17:05   max work group size 1024
2019-08-01 10:17:05   max work item sizes [1024, 1024, 64]
2019-08-01 10:17:05 Using CUDA device 'GeForce RTX 2070' in concurrent. (2 times)
2019-08-01 10:17:05 Using CUDA device 'GeForce RTX 2070' in concurrent. (2 times)
2019-08-01 10:17:05 Initializing...
2019-08-01 10:17:05 sorting point cloud... done in 0.554 sec
2019-08-01 10:17:06 processing matches... done in 0.145 sec
2019-08-01 10:17:06 initializing...
2019-08-01 10:17:20 selected 801 cameras from 1482 in 13.787 sec
2019-08-01 10:17:22 Loading photos...

....
 Depth reconstruction devices performance:
  - 47%    done by GeForce RTX 2070
  - 53%    done by GeForce RTX 2070
 Total time: 2780.5 seconds
 
 Generating dense point cloud...
 Generating dense point cloud...
 selected 392 cameras in 7.514 sec
 working volume: 10826x5861x8361
 tiles: 3x1x2
 selected 109 cameras in 0.001 sec
 using 109 depth maps including neighbors (7.88121 GB)
 preloading data... done in 7.566 sec
 filtering 109 depth maps...
 filtering DSC05502 using 74 neighbors... done in 9.668 sec (load: 0.922, render: 1.94 filter: 6.806)
 filtering DSC05503 using 79 neighbors... done in 12.219 sec (load: 0.993, render: 2.191 filter: 9.035)
 filtering DSC06346 using 77 neighbors... done in 6.277 sec (load: 0.975, render: 2.767 filter: 2.535)
 filtering DSC06347 using 76 neighbors... done in 6.699 sec (load: 0.98, render: 2.665 filter: 3.054)
 filtering DSC06348 using 71 neighbors... done in 5.434 sec (load: 0.928, render: 1.977 filter: 2.529)
...
 filtering DSC05531 using 100 neighbors... done in 10.75 sec (load: 1.019, render: 2.009 filter: 7.722)
 filtering DSC05532 using 100 neighbors... done in 11.764 sec (load: 1.026, render: 2.008 filter: 8.73)
 filtering DSC05533 using 100 neighbors...

--> here Metashape was terminated without an Error-Message.

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14813
    • View Profile
Re: Hard system crash consistently during depth map filtering
« Reply #2 on: August 08, 2019, 01:31:24 PM »
Hello Dannosuke,

Can you please share the processing log from the procedure start till the computer lock up?


Hello 3DFranz,

Please check, if there are any "error" marked lines in Windows Event Viewer (Windows Logs -> Application section) related to metashape.exe. If there are any, please specify the signature.
The problem could be related to the RAM issues, so I could also suggest to run Memtest86 checks in multithreaded mode.
If the system is overclocked, it may be also a reason of the problem.
Best regards,
Alexey Pasumansky,
Agisoft LLC

saduka

  • Newbie
  • *
  • Posts: 33
    • View Profile
Re: Hard system crash consistently during depth map filtering
« Reply #3 on: August 08, 2019, 01:53:08 PM »
Hi everyone,

I am having a consistent problem. My computer locks up with a black screen during the depth map filtering process which requires a hard reset. I have no problem running any other program or bench marking program. But every time I try to create a dense point cloud the computer locks up.  It lets me create a sparse point cloud no problem. I'm running Metashape on a Threadripper 2990wx processor, with two Radon VII GPUs, and 128 GB of RAM.

thanks,

Hi 3DFranz
I have met same problems several times during the adaption from PhotoScan to Metashape, too.
But when I specified the newly introduced argument  "max_neighbours=50", problems were mysteriously solved. I doubt theres some default parameters not well set here.

3DFranz

  • Newbie
  • *
  • Posts: 44
    • View Profile
Re: Hard system crash consistently during depth map filtering
« Reply #4 on: August 09, 2019, 10:57:48 AM »
Dear Alexey,

indeed it was a overclocking problem.
Caused by Thundermaster from Palit.
Thank you for your help.
Best reagrds,
Franz

Dannosuke

  • Newbie
  • *
  • Posts: 13
    • View Profile
Re: Hard system crash consistently during depth map filtering
« Reply #5 on: August 28, 2019, 12:59:09 AM »
Hello Dannosuke,

Can you please share the processing log from the procedure start till the computer lock up?


Hello 3DFranz,

Please check, if there are any "error" marked lines in Windows Event Viewer (Windows Logs -> Application section) related to metashape.exe. If there are any, please specify the signature.
The problem could be related to the RAM issues, so I could also suggest to run Memtest86 checks in multithreaded mode.
If the system is overclocked, it may be also a reason of the problem.

Hi Alexey,

I no longer am having a hard system crash but the program is still crashing and shutting down leaving the computer system operational.

here is the log

Thanks for any help you can offer,

Dan

ARB_texture_non_power_of_two: Yes
ARB_vertex_buffer_object: Yes
CPU: AMD Ryzen Threadripper 2990WX 32-Core Processor
OpenGLMaxTextureSize: 16384
OpenGLRenderer: AMD Radeon VII
OpenGLStereo: No
OpenGLVendor: ATI Technologies Inc.
OpenGLVersion: 4.6.13570 Compatibility Profile Context 19.7.5 26.20.13001.25001
ProductName: Metashape Pro
System: Windows 64bit
SystemMemory: 127.9 GB
Throttleable: 1
URL:
Vendor: Agisoft
Version: 1.5.2.7838
XLog: GPU] estimating 1569x1039x448 disparity using 785x1056x8u tiles
[GPU] estimating 1707x1234x736 disparity using 854x1248x8u tiles
[GPU] estimating 1661x1260x480 disparity using 831x1280x8u tiles
[GPU] estimating 1673x1294x288 disparity using 837x1312x8u tiles
timings: rectify: 0.026 disparity: 0.195 borders: 0.018 filter: 0.068 fill: 0
timings: rectify: 0.02 disparity: 0.215 borders: 0.032 filter: 0.089 fill: 0
timings: rectify: 0.025 disparity: 0.243 borders: 0.033 filter: 0.076 fill: 0
[GPU] estimating 1656x1277x256 disparity using 828x1280x8u tiles
[GPU] estimating 1672x1254x480 disparity using 836x1280x8u tiles
...
[GPU] estimating 1621x1242x288 disparity using 811x1248x8u tiles
timings: rectify: 0.025 disparity: 0.197 borders: 0.031 filter: 0.084 fill: 0
timings: rectify: 0.026 disparity: 0.208 borders: 0.034 filter: 0.08 fill: 0
[GPU] estimating 1675x1250x192 disparity using 838x1280x8u tiles
timings: rectify: 0.019 disparity: 0.182 borders: 0.024 filter: 0.079 fill: 0
[GPU] estimating 1661x1260x480 disparity using 831x1280x8u tiles
timings: rectify: 0.027 disparity: 0.184 borders: 0.026 filter: 0.08 fill: 0
[GPU] estimating 1548x1139x544 disparity using 774x1152x8u tiles
timings: rectify: 0.019 disparity: 0.2 borders: 0.023 filter: 0.079 fill: 0
[GPU] estimating 780x1721x416 disparity using 780x864x8u tiles
timings: rectify: 0.022 disparity: 0.153 borders: 0.02 filter: 0.068 fill: 0
[GPU] estimating 1468x1264x448 disparity using 1468x1280x8u tiles
[GPU] estimating 1529x1267x256 disparity using 1529x1280x8u tiles
timings: rectify: 0.02 disparity: 0.13 borders: 0.019 filter: 0.061 fill: 0
[GPU] estimating 1647x1244x256 disparity using 824x1248x8u tiles
timings: rectify: 0.016 disparity: 0.208 borders: 0.019 filter: 0.108 fill: 0
timings: rectify: 0.029 disparity: 0.217 borders: 0.041 filter: 0.08 fill: 0


This report also contains technical information about the state of the application when it crashed.
« Last Edit: August 28, 2019, 01:00:40 AM by Dannosuke »

Dannosuke

  • Newbie
  • *
  • Posts: 13
    • View Profile
Re: Hard system crash consistently during depth map filtering
« Reply #6 on: August 29, 2019, 04:25:19 AM »
Now I'm back to hard crashing again

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14813
    • View Profile
Re: Hard system crash consistently during depth map filtering
« Reply #7 on: August 30, 2019, 02:45:46 PM »
Hello Dannosuke,

Can you please locate the "error" marked lines related to metashape.exe in Windows Event Viewer -> Windows Logs -> Application section, export them to .evtx file and send to support@agisoft.com?

Also I can suggest to check, if you are still observing the crashes with the following Tweak set to value 1 (via Advanced Preferences tab):
Code: [Select]
main/depth_max_gpu_multiplierBoth when all the GPUs are used and when only one is enabled in GPU Preferences tab.
Best regards,
Alexey Pasumansky,
Agisoft LLC

Dannosuke

  • Newbie
  • *
  • Posts: 13
    • View Profile
Re: Hard system crash consistently during depth map filtering
« Reply #8 on: October 17, 2019, 04:33:13 AM »
Hi Alexey, After I turned off one of my GPUs the crashing stopped.  But now it crashes every time I try to import masks it will do a few then the system locks up and the and the screen turns off. The only way to get it going again is hitting the reset button to restart.

here is the log it generated.

2019-10-16 18:18:37 Loading project...
2019-10-16 18:18:41 loaded project in 4.141 sec
2019-10-16 18:18:41 Finished processing in 4.144 sec (exit code 1)
2019-10-16 18:19:47 ImportMasks: cameras = 0, method = MaskSourceBackground, path = E:/Google Drive/Dissertation/Coral Reef Research/Tank Experiments/Heron Is. Data/3D Models/Photos/MO4/Mask.jpg, tolerance = 22
2019-10-16 18:19:47 Generating masks...
2019-10-16 18:19:49 Finished processing in 1.947 sec (exit code 1)
2019-10-16 18:20:12 Updating point cloud...
2019-10-16 18:20:12 disabled 1676 points
2019-10-16 18:20:12 Finished processing in 0.009 sec (exit code 1)
2019-10-16 18:20:34 ImportMasks: method = MaskSourceBackground, path = E:/Google Drive/Dissertation/Coral Reef Research/Tank Experiments/Heron Is. Data/3D Models/Photos/MO4/mask.jpg, tolerance = 22
2019-10-16 18:20:34 Generating masks...
2019-10-16 18:20:43 Finished processing in 9.313 sec (exit code 1)
2019-10-16 18:21:05 ImportMasks: method = MaskSourceBackground, path = E:/Google Drive/Dissertation/Coral Reef Research/Tank Experiments/Heron Is. Data/3D Models/Photos/MO4/mask.jpg, tolerance = 22
2019-10-16 18:21:05 Generating masks...

Crashes here

Not sure how to get logs out of windows event viewer, but after looking at it I don't see anything about Metashape

Any help here would be great.  Having to manually set masks one by one is a real time suck and if I could just get it to import then that would be awesome

Also, my CPU debug light on the motherboard lights up and a debug code 00 shows as well.  That code doesn't show up in my motherboard manual. I have the MSI X399 Carbon/AC Gaming Pro with a Threadripper 2990WX CPU

Thanks!
« Last Edit: October 17, 2019, 04:45:03 AM by Dannosuke »

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14813
    • View Profile
Re: Hard system crash consistently during depth map filtering
« Reply #9 on: October 17, 2019, 11:06:36 AM »
Hello Dannosuke,

Can you please send the example of the single image and a background image used to support@agisoft.com, so that we could try to reproduce the problem on our side?

If you see the crash reporter window, please also submit the report.

As for the original issue with the depth maps generation, it looks like the latest AMD driver should fix the issue, so if you have some spare time for checking that, I can suggest to enable AMD cards again, set the main/depth_max_gpu_multiplier tweak value to 2 and try to build the depth maps again.
Best regards,
Alexey Pasumansky,
Agisoft LLC

Costas

  • Jr. Member
  • **
  • Posts: 67
  • Aerial Mapping
    • View Profile
Re: Hard system crash consistently during depth map filtering
« Reply #10 on: October 18, 2019, 03:04:05 PM »
I had the same problem in the past. (AMD CPU)
Did all tests for memory, cpu, gpu etc with no problem. Finally i found the problem in BIOS.

The problem was overclocking. I entered the BIOS (UEFI) settings and turned off any overclocking in CPU and Memory. Running normal speeds with XMP memory profile solved the problem and never had a crash after that.

Dannosuke

  • Newbie
  • *
  • Posts: 13
    • View Profile
Re: Hard system crash consistently during depth map filtering
« Reply #11 on: October 19, 2019, 02:21:01 AM »
Costa, I thought the same thing so I set my BIOS back to optimized Defaults. And the same thing happened.

Alexey I'll send that over to you when i get a chance.

Thanks!