Forum

Author Topic: Not enough Memory - yet memory visibly not being used in resource monitor  (Read 3952 times)

solosails

  • Newbie
  • *
  • Posts: 13
    • View Profile
Hi,

Much asked question, I have read all the threads on this subject, but none seem to mention like me that the memory is not actually being used anyway near its full potential.

I only have 32GB RAM installed on a windows 10 Pro (so can use all of RAM) OS, yet I keep getting "out of memory" errors, yet resource monitor shows that there is significantly over half the RAM free at all times.

I had been using GPU processing with use CPU and GPU compute option checked, I am now trying Just CPU in case it is a GPU RAM issue, not the PC, the GPU is only 2GB memory, so this could be the issue perhaps? its a GTX 680

Currently running the same problematic run right now and metashape is only using 2GB of memory whilst building mesh - yet it will fail on memory.

Any ideas? Thanks.

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 13300
    • View Profile
Hello solosails,

Can you please provide the processing log from the node where the "out of memory" message occurred?
Best regards,
Alexey Pasumansky,
Agisoft LLC

solosails

  • Newbie
  • *
  • Posts: 13
    • View Profile
Hi Alexey,

Thanks for the reply. I'm just running it right now using CPU only, when it finishes the run (or errors out) I'll send the log.

solosails

  • Newbie
  • *
  • Posts: 13
    • View Profile
Hello solosails,

Can you please provide the processing log from the node where the "out of memory" message occurred?

OK, so it completed using CPU only (however, I used lower settings, so maybe this info isnt helpfull - but here is the log for the failed GPU/CPU run....

019-01-02 15:33:03 BuildDenseCloud: quality = High, depth filtering = Mild, reuse depth maps, point colors = 1
2019-01-02 15:33:03 Generating dense point cloud...
2019-01-02 15:33:03 Generating dense point cloud...
2019-01-02 15:33:03 initializing...
2019-01-02 15:33:09 selected 186 cameras in 6.218 sec
2019-01-02 15:33:09 working volume: 2543x2402x1729
2019-01-02 15:33:09 tiles: 1x1x1
2019-01-02 15:33:09 saved dense cloud data in 0.004 sec
2019-01-02 15:33:09 saved regions in 0.008 sec
2019-01-02 15:33:09 saved camera partition in 0.007 sec
2019-01-02 15:33:09 scheduled 10 depth filtering groups
2019-01-02 15:33:09 scheduled 1 dense cloud regions
2019-01-02 15:33:09 loaded camera partition in 0 sec
2019-01-02 15:33:09 loaded dense cloud data in 0.001 sec
2019-01-02 15:33:10 preloading data... done in 8.702 sec
2019-01-02 15:33:19 filtering depth maps... Finished processing in 1224.39 sec (exit code 0)
2019-01-02 15:53:27 Error: Aborted by user
2019-01-02 15:54:10 Analyzing dense cloud...
2019-01-02 15:54:11 Finished processing in 1.236 sec (exit code 1)
2019-01-02 15:54:34 BuildModel: source data = Depth maps, surface type = Arbitrary, quality = High, depth filtering = Mild, face count = Medium, smoothness = Low, vertex colors = 1
2019-01-02 15:54:35 Generating mesh...
2019-01-02 15:54:35 [GPU 1] Using device: GeForce GTX 680, 8 compute units, free memory: 1627/2048 MB, compute capability 3.0
2019-01-02 15:54:35   driver version: 417.35, driver/runtime CUDA: 10000/5050
2019-01-02 15:54:35   max work group size 1024
2019-01-02 15:54:35   max work item sizes [1024, 1024, 64]
2019-01-02 15:54:35 Preparing depth maps...
2019-01-02 15:54:35 Using 238 cameras
2019-01-02 15:57:43   Depth maps: 186
2019-01-02 15:57:43   done in 188.507 sec
2019-01-02 15:57:43 Subdividing space...
2019-01-02 15:59:04   Nodes: 255546177
2019-01-02 15:59:04   done in 80.374 sec
2019-01-02 15:59:04 Balancing space...
2019-01-02 16:03:19   Nodes: 296923625
2019-01-02 16:03:19   done in 255.537 sec
2019-01-02 16:03:19 Estimating density...
2019-01-02 16:04:43   done in 83.573 sec
2019-01-02 16:04:43 Estimating surface...
2019-01-02 16:04:45 Level #5/13
2019-01-02 16:04:45   Subdividing...
2019-01-02 16:04:45     done in 0.514 sec
2019-01-02 16:04:46 [GPU 1] 1 Parts 1-1/1...
2019-01-02 16:04:46 [GPU 1]   Initializing octree...
2019-01-02 16:04:46 [GPU 1]   Initializing histograms...
2019-01-02 16:05:17 [GPU 1]   Initializing to most probable...
2019-01-02 16:05:17 [GPU 1]   Iterating...
2019-01-02 16:05:17 [GPU 1]   Updating result...
2019-01-02 16:05:19 [GPU 1] Part done in 33.591 sec
2019-01-02 16:05:19 Devices performance:
2019-01-02 16:05:19  - 100%    done by [GPU 1]  with timings: 33 s = 0% fetch + 2% init + 91% hist (0% GPU) + 0% init + 0% iters (65% + 34%) + 6% updating
2019-01-02 16:05:19 IO utilization: 5%
2019-01-02 16:05:19 Level done in 34.304 sec
2019-01-02 16:05:19 Level #7/13
2019-01-02 16:05:19   Subdividing...
2019-01-02 16:05:20     done in 0.479 sec
2019-01-02 16:05:20 [GPU 1] 1 Parts 1-1/1...
2019-01-02 16:05:20 [GPU 1]   Initializing octree...
2019-01-02 16:05:21 [GPU 1]   Initializing histograms...
2019-01-02 16:05:49 [GPU 1]   Initializing...
2019-01-02 16:05:51 [GPU 1]   Iterating...
2019-01-02 16:05:54 [GPU 1]   Updating result...
2019-01-02 16:05:56 [GPU 1] Part done in 35.832 sec
2019-01-02 16:05:56 Devices performance:
2019-01-02 16:05:56  - 100%    done by [GPU 1]  with timings: 35 s = 0% fetch + 2% init + 81% hist (5% GPU) + 3% init + 8% iters (82% + 18%) + 6% updating
2019-01-02 16:05:56 IO utilization: 5%
2019-01-02 16:05:56 Level done in 36.501 sec
2019-01-02 16:05:56 Level #9/13
2019-01-02 16:05:56   Subdividing...
2019-01-02 16:05:56     done in 0.631 sec
2019-01-02 16:05:56 [GPU 1] 6 Parts 1-6/8...
2019-01-02 16:05:56 [GPU 1]   Initializing octree...
2019-01-02 16:06:00 [GPU 1]   Initializing histograms...
2019-01-02 16:06:36 [GPU 1]   Initializing...
2019-01-02 16:06:38 [GPU 1]   Iterating...
2019-01-02 16:07:34 [GPU 1]   Updating result...
2019-01-02 16:07:36 [GPU 1] Part done in 99.912 sec
2019-01-02 16:07:36 [GPU 1] 2 Parts 7-8/8...
2019-01-02 16:07:36 [GPU 1]   Initializing octree...
2019-01-02 16:07:37 [GPU 1]   Initializing histograms...
2019-01-02 16:08:05 [GPU 1]   Initializing...
2019-01-02 16:08:06 [GPU 1]   Iterating...
2019-01-02 16:08:10 [GPU 1]   Updating result...
2019-01-02 16:08:10 [GPU 1] Part done in 33.731 sec
2019-01-02 16:08:10 Devices performance:
2019-01-02 16:08:10  - 100%    done by [GPU 1]  with timings: 133 s = 0% fetch + 3% init + 47% hist (49% GPU) + 2% init + 44% iters (83% + 17%) + 3% updating
2019-01-02 16:08:10 IO utilization: 3%
2019-01-02 16:08:10 Level done in 134.487 sec
2019-01-02 16:08:10 Level #11/13
2019-01-02 16:08:10   Subdividing...
2019-01-02 16:08:12     done in 1.57 sec
2019-01-02 16:08:12 [GPU 1] 5 Parts 1-5/148...
2019-01-02 16:08:12 [GPU 1]   Initializing octree...
2019-01-02 16:08:17 [GPU 1]   Initializing histograms...
2019-01-02 16:12:00 [GPU 1]   Initializing...
2019-01-02 16:12:03 [GPU 1]   Iterating...
2019-01-02 16:19:49 [GPU 1]   Updating result...
2019-01-02 16:19:50 [GPU 1] Part done in 697.904 sec
2019-01-02 16:19:50 [GPU 1] 9 Parts 6-14/148...
2019-01-02 16:19:50 [GPU 1]   Initializing octree...
2019-01-02 16:19:55 [GPU 1]   Initializing histograms...
2019-01-02 16:23:42 [GPU 1]   Initializing...
2019-01-02 16:23:45 [GPU 1]   Iterating...
2019-01-02 16:31:21 [GPU 1]   Updating result...
2019-01-02 16:31:23 [GPU 1] Part done in 692.748 sec
2019-01-02 16:31:23 [GPU 1] 5 Parts 15-19/148...
2019-01-02 16:31:23 [GPU 1]   Initializing octree...
2019-01-02 16:31:28 [GPU 1]   Initializing histograms...
2019-01-02 16:33:57 [GPU 1]   Initializing...
2019-01-02 16:33:59 [GPU 1]   Iterating...
2019-01-02 16:36:45 [GPU 1]   Updating result...
2019-01-02 16:36:47 [GPU 1] Part done in 324.016 sec
2019-01-02 16:36:47 [GPU 1] 5 Parts 20-24/148...
2019-01-02 16:36:47 [GPU 1]   Initializing octree...
2019-01-02 16:36:53 [GPU 1]   Initializing histograms...
2019-01-02 16:39:04 [GPU 1]   Initializing...
2019-01-02 16:39:06 [GPU 1]   Iterating...
2019-01-02 16:43:33 [GPU 1]   Updating result...
2019-01-02 16:43:37 [GPU 1] Part done in 410.484 sec
2019-01-02 16:43:37 [GPU 1] 7 Parts 25-31/148...
2019-01-02 16:43:37 [GPU 1]   Initializing octree...
2019-01-02 16:43:45 [GPU 1]   Initializing histograms...
2019-01-02 16:45:29 [GPU 1]   Initializing...
2019-01-02 16:45:31 [GPU 1]   Iterating...
2019-01-02 16:46:37 [GPU 1]   Updating result...
2019-01-02 16:46:38 [GPU 1] Part done in 181.268 sec
2019-01-02 16:46:38 [GPU 1] 6 Parts 32-37/148...
2019-01-02 16:46:38 [GPU 1]   Initializing octree...
2019-01-02 16:46:43 [GPU 1]   Initializing histograms...
2019-01-02 16:49:03 [GPU 1]   Initializing...
2019-01-02 16:49:05 [GPU 1]   Iterating...
2019-01-02 16:53:36 [GPU 1]   Updating result...
2019-01-02 16:53:38 [GPU 1] Part done in 419.143 sec
2019-01-02 16:53:38 [GPU 1] 8 Parts 38-45/148...
2019-01-02 16:53:38 [GPU 1]   Initializing octree...
2019-01-02 16:53:43 [GPU 1]   Initializing histograms...
2019-01-02 16:55:45 [GPU 1]   Initializing...
2019-01-02 16:55:48 [GPU 1]   Iterating...
2019-01-02 16:59:41 [GPU 1]   Updating result...
2019-01-02 16:59:42 [GPU 1] Part done in 364.535 sec
2019-01-02 16:59:42 [GPU 1] 8 Parts 46-53/148...
2019-01-02 16:59:42 [GPU 1]   Initializing octree...
2019-01-02 16:59:48 [GPU 1]   Initializing histograms...
2019-01-02 17:01:20 [GPU 1]   Initializing...
2019-01-02 17:01:23 [GPU 1]   Iterating...
2019-01-02 17:04:11 [GPU 1]   Updating result...
2019-01-02 17:04:13 [GPU 1] Part done in 270.713 sec
2019-01-02 17:04:13 [GPU 1] 8 Parts 54-61/148...
2019-01-02 17:04:13 [GPU 1]   Initializing octree...
2019-01-02 17:04:18 [GPU 1]   Initializing histograms...
2019-01-02 17:06:38 [GPU 1]   Initializing...
2019-01-02 17:06:40 [GPU 1]   Iterating...
2019-01-02 17:11:07 [GPU 1]   Updating result...
2019-01-02 17:11:08 [GPU 1] Part done in 415.087 sec
2019-01-02 17:11:08 [GPU 1] 9 Parts 62-70/148...
2019-01-02 17:11:08 [GPU 1]   Initializing octree...
2019-01-02 17:11:13 [GPU 1]   Initializing histograms...
2019-01-02 17:12:45 [GPU 1]   Initializing...
2019-01-02 17:12:47 [GPU 1]   Iterating...
2019-01-02 17:15:33 [GPU 1]   Updating result...
2019-01-02 17:15:34 [GPU 1] Part done in 266.279 sec
2019-01-02 17:15:34 [GPU 1] 11 Parts 71-81/148...
2019-01-02 17:15:34 [GPU 1]   Initializing octree...
2019-01-02 17:15:39 [GPU 1]   Initializing histograms...
2019-01-02 17:16:18 [GPU 1]   Initializing...
2019-01-02 17:16:20 [GPU 1]   Iterating...
2019-01-02 17:17:25 [GPU 1]   Updating result...
2019-01-02 17:17:26 [GPU 1] Part done in 111.885 sec
2019-01-02 17:17:26 [GPU 1] 12 Parts 82-93/148...
2019-01-02 17:17:26 [GPU 1]   Initializing octree...
2019-01-02 17:17:31 [GPU 1]   Initializing histograms...
2019-01-02 17:18:15 [GPU 1]   Initializing...
2019-01-02 17:18:17 [GPU 1]   Iterating...
2019-01-02 17:19:21 [GPU 1]   Updating result...
2019-01-02 17:19:22 [GPU 1] Part done in 116.118 sec
2019-01-02 17:19:22 [GPU 1] 17 Parts 94-110/148...
2019-01-02 17:19:22 [GPU 1]   Initializing octree...
2019-01-02 17:19:28 [GPU 1]   Initializing histograms...
2019-01-02 17:20:09 [GPU 1]   Initializing...
2019-01-02 17:20:11 [GPU 1]   Iterating...
2019-01-02 17:21:15 [GPU 1]   Updating result...
2019-01-02 17:21:16 [GPU 1] Part done in 113.81 sec
2019-01-02 17:21:16 [GPU 1] 38 Parts 111-148/148...
2019-01-02 17:21:16 [GPU 1]   Initializing octree...
2019-01-02 17:21:20 [GPU 1]   Initializing histograms...
2019-01-02 17:21:49 [GPU 1]   Initializing...
2019-01-02 17:21:51 [GPU 1]   Iterating...
2019-01-02 17:22:22 [GPU 1]   Updating result...
2019-01-02 17:22:23 [GPU 1] Part done in 66.767 sec
2019-01-02 17:22:23 Devices performance:
2019-01-02 17:22:23  - 100%    done by [GPU 1]  with timings: 4451 s = 0% fetch + 1% init + 35% hist (95% GPU) + 1% init + 62% iters (75% + 25%) + 0% updating
2019-01-02 17:22:23 IO utilization: 2%
2019-01-02 17:22:23 Level done in 4452.93 sec
2019-01-02 17:22:23 Level #13/13
2019-01-02 17:22:23   Subdividing...
2019-01-02 17:22:25     done in 2.234 sec
2019-01-02 17:22:25 [GPU 1] 5 Parts 1-5/316...
2019-01-02 17:22:25 [GPU 1]   Initializing octree...
2019-01-02 17:22:31 [GPU 1]   Initializing histograms...
2019-01-02 17:22:32 Finished processing in 5277.28 sec (exit code 0)
2019-01-02 17:22:32 Error: out of memory (2) at line 179

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 13300
    • View Profile
Hello solosails,

"Out of memory" message is related to GPU processing.

Can you please specify if you have the latest NVIDIA driver version for your GPU installed, and which version (including the build number) of Metashape you are using?
Best regards,
Alexey Pasumansky,
Agisoft LLC

solosails

  • Newbie
  • *
  • Posts: 13
    • View Profile
Hi Alexey,

Thanks for analysing the log.

Yes, I have the latest Nvidia driver (currently - 417.35)

Metashape V 1.5.0 build 7492 64bit