Forum

Author Topic: Not enough memory on GPU  (Read 4287 times)

Dieter

  • Full Member
  • ***
  • Posts: 198
    • View Profile
Not enough memory on GPU
« on: August 31, 2021, 07:14:54 PM »
I want to texture and/or building tiled modell.

I always have this message in my console:


Estimated required video memory: XXXXX MB
Estimated device memory: total 4043 MB, used 65 MB, available 3437 MB
Cannot use GPU. Reason: Not enough memory on device


I always thought that metashape divides the model  according to the GPU?

Tested with GTX 1060 and GTX 970, my nvidia driver is 471.68 working on Metashabe 17.4

What can I do to calculate textures on the GPU?

Thanks
« Last Edit: August 31, 2021, 07:21:50 PM by Dieter »

bgreenstone

  • Jr. Member
  • **
  • Posts: 97
    • View Profile
Re: Not enough memory on GPU
« Reply #1 on: September 02, 2021, 06:46:29 PM »
I was having a similar problem on my old Mac Pro earlier this year, and when I contacted Agisoft they sent me this:

-----------

Please try to create the BuildDepthMaps/pm_ooc_ref_tile_size tweak via Advanced preferences tab and set its value to 512 (default is 2048),

-----------

That mostly fixed the issue for me.

-Brian

Dieter

  • Full Member
  • ***
  • Posts: 198
    • View Profile
Re: Not enough memory on GPU
« Reply #2 on: September 04, 2021, 06:28:16 AM »
Thank you for the reply, but there is no change with this parameter.

Alexey, can you help me??

Dieter

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14813
    • View Profile
Re: Not enough memory on GPU
« Reply #3 on: September 07, 2021, 10:17:39 PM »
Hello Dieter,

Can you provide longer part of the log which comes before the not enough memory message?

Also specify the tiled model reconstruction parameters and number/resolution of the images in the project.
Best regards,
Alexey Pasumansky,
Agisoft LLC

Dieter

  • Full Member
  • ***
  • Posts: 198
    • View Profile
Re: Not enough memory on GPU
« Reply #4 on: September 09, 2021, 08:56:35 AM »
Hello Alexey, here is the log:

Avg camera fetch time: 0.242729 sec
Total stalled time: 0.264 sec
Avg camera processing time: 1.21016 sec
Processed cameras
postprocessing atlas... done in 0.664 sec
applying textures... done in 0.056 sec
saved textures in 0.099 sec
loaded tiled model data in 0 sec
loaded block index in 0.001 sec
loaded block index in 0 sec
loaded block index in 0 sec
loaded camera partition in 0.004 sec
Found 2 GPUs in 0 sec (CUDA: 0 sec, OpenCL: 0 sec)
Using device: NVIDIA GeForce GTX 1060 6GB, 10 compute units, free memory: 5169/6144 MB, compute capability 6.1
  driver/runtime CUDA: 11040/8000
  max work group size 1024
  max work item sizes [1024, 1024, 64]
loaded selector in 0.001 sec
selected 815 cameras in 19.586 sec
selected 835 blocks in 0.878 sec
loaded model blocks in 0.566 sec
loaded uv blocks in 0.398 sec
initializing renderer... done in 0.486 sec
Blending textures...
All cameras are fine
Activating context...
calculating mesh connectivity... done in 2.156 sec
rendering 0 to 100 pages
Initialized texture renderer
Configuring pipeline...
Initialized memory broker
Configuring mosaic pipeline with outliers filtering
Constructed pipeline
Relaxed precision enabled
Collecting memory requests...
Allocating memory requests...
Estimated required video memory: 4016 MB
Estimated device memory: total 6052 MB, used 114 MB, available 5284 MB
Allocated 4033 MB of VRAM
Allocated memory requests
Configured pipeline
Processing cameras...
Loading mesh...
viewport estimation started
*************************************************************************************************
viewport estimation done. (in 3.402 sec)
Avg camera fetch time: 1.59509e-05 sec
Total stalled time: 0.003 sec
Avg camera processing time: 0.00415215 sec
resolution disparity estimation started
**************************************************************************************************
resolution disparity estimation done. (in 71.026 sec)
Avg camera fetch time: 8.58896e-06 sec
Total stalled time: 0.003 sec
Avg camera processing time: 0.0871018 sec
outliers detection started
***************************************************************************************************
outliers detection done. (in 301.963 sec)
Avg camera fetch time: 0.028946 sec
Total stalled time: 0.024 sec
Avg camera processing time: 0.370429 sec
Compressed texture pages. (in 9.955 sec)
Compressed texture pages. (in 10.031 sec)
texture detail/outlier estimation started
***************************************************************************************************
texture detail/outlier estimation done. (in 533.348 sec)
Avg camera fetch time: 0.0157215 sec
Total stalled time: 0.01 sec
Avg camera processing time: 0.654342 sec
final blending/outliers started
****************************************************************************************************
final blending/outliers done. (in 996.091 sec)
Avg camera fetch time: 0.260971 sec
Total stalled time: 0.215 sec
Avg camera processing time: 1.22188 sec
Processed cameras
postprocessing atlas... done in 2.911 sec
applying textures... done in 0.14 sec
selected 768 cameras in 12.238 sec
selected 780 blocks in 2.708 sec
loaded model blocks in 0.682 sec
loaded uv blocks in 0.694 sec
initializing renderer... done in 0.49 sec
Blending textures...
All cameras are fine
Activating context...
calculating mesh connectivity... done in 2.033 sec
rendering 0 to 39 pages
Initialized texture renderer
Configuring pipeline...
Initialized memory broker
Configuring mosaic pipeline with outliers filtering
Constructed pipeline
Relaxed precision enabled
Collecting memory requests...
Allocating memory requests...
Estimated required video memory: 2506 MB
Estimated device memory: total 6052 MB, used 114 MB, available 5284 MB
Allocated 2523 MB of VRAM
Allocated memory requests
Configured pipeline
Processing cameras...
Loading mesh...
viewport estimation started
****************************************************************************************************
viewport estimation done. (in 3.082 sec)
Avg camera fetch time: 1.17187e-05 sec
Total stalled time: 0.001 sec
Avg camera processing time: 0.00399089 sec
resolution disparity estimation started
****************************************************************************************************
resolution disparity estimation done. (in 63.144 sec)
Avg camera fetch time: 1.43229e-05 sec
Total stalled time: 0.002 sec
Avg camera processing time: 0.0821875 sec
outliers detection started
*****************************************************************************************************
outliers detection done. (in 271.309 sec)
Avg camera fetch time: 0.0275299 sec
Total stalled time: 0.018 sec
Avg camera processing time: 0.353186 sec
Compressed texture pages. (in 3.87 sec)
Compressed texture pages. (in 3.717 sec)
texture detail/outlier estimation started
*****************************************************************************************************
texture detail/outlier estimation done. (in 475.747 sec)
Avg camera fetch time: 0.0154531 sec
Total stalled time: 0.019 sec
Avg camera processing time: 0.619371 sec
final blending/outliers started
******************************************************************************************************
final blending/outliers done. (in 888.072 sec)
Avg camera fetch time: 0.250829 sec
Total stalled time: 0.279 sec
Avg camera processing time: 1.15592 sec
Processed cameras
postprocessing atlas... done in 1.27 sec
applying textures... done in 0.072 sec
selected 571 cameras in 13.756 sec
selected 711 blocks in 0.669 sec
loaded model blocks in 0.481 sec
loaded uv blocks in 0.391 sec
initializing renderer... done in 0.44 sec
Blending textures...
All cameras are fine
Activating context...
calculating mesh connectivity... done in 1.785 sec
rendering 0 to 31 pages
Initialized texture renderer
Configuring pipeline...
Initialized memory broker
Configuring mosaic pipeline with outliers filtering
Constructed pipeline
Relaxed precision enabled
Collecting memory requests...
Allocating memory requests...
Estimated required video memory: 2244 MB
Estimated device memory: total 6052 MB, used 114 MB, available 5284 MB
Allocated 2261 MB of VRAM
Allocated memory requests
Configured pipeline
Processing cameras...
Loading mesh...
viewport estimation started
***************************************************************************************************
viewport estimation done. (in 2.049 sec)
Avg camera fetch time: 1.05079e-05 sec
Total stalled time: 0.002 sec
Avg camera processing time: 0.00356217 sec
resolution disparity estimation started
**************************************************************************************************
resolution disparity estimation done. (in 41.652 sec)
Avg camera fetch time: 5.07881e-05 sec
Total stalled time: 0.01 sec
Avg camera processing time: 0.0728301 sec
outliers detection started
***************************************************************************************************
outliers detection done. (in 178.338 sec)
Avg camera fetch time: 0.0201086 sec
Total stalled time: 0.016 sec
Avg camera processing time: 0.312158 sec
Compressed texture pages. (in 2.37 sec)
Compressed texture pages. (in 2.393 sec)
texture detail/outlier estimation started
**************************************************************************************************
texture detail/outlier estimation done. (in 320.234 sec)
Avg camera fetch time: 0.019662 sec
Total stalled time: 0.012 sec
Avg camera processing time: 0.560664 sec
final blending/outliers started
**************************************************************************************************
final blending/outliers done. (in 553.403 sec)
Avg camera fetch time: 0.243933 sec
Total stalled time: 0.112 sec
Avg camera processing time: 0.968837 sec
Processed cameras
postprocessing atlas... done in 1.022 sec
applying textures... done in 0.053 sec
selected 1784 cameras in 18.249 sec
selected 1310 blocks in 2.842 sec
loaded model blocks in 1.576 sec
loaded uv blocks in 1.613 sec
initializing renderer... done in 0.734 sec
Blending textures...
All cameras are fine
Activating context...
calculating mesh connectivity... done in 3.199 sec
rendering 0 to 137 pages
Initialized texture renderer
Configuring pipeline...
Initialized memory broker
Configuring mosaic pipeline with outliers filtering
Constructed pipeline
Relaxed precision enabled
Collecting memory requests...
Allocating memory requests...
Estimated required video memory: 5223 MB
Estimated device memory: total 6052 MB, used 114 MB, available 5284 MB
Cannot use GPU. Reason: Not enough memory on device
Performing blending on CPU...

rendering 137 pages
calculating mesh connectivity... done in 3.263 sec
detecting outliers... ***?**************************************************************************************************** done in 5578.94 sec
estimating quality... ***?*****************************************************************************************************


As you can see: sometimes it works, sometimes not....

The settings were as follows:
Tiled Model:
7004 cameras with 20 mpix
pixel size 0.002!!  (This is a requirement of the client)
tile size 1024
Source: Mesh
Face count: medium
Ghosting filter: on

With both GPUs (GTX1060 or GTX970), it looks like Metashape is calculating the GPU's free memory too generously. A tilesize of 512 did not bring any improvement either. Sometimes the creation of the model runs up to 60% only on the GPU, then the memory is calculated incorrectly and it is only CPU, which is slower by a factor of 10 for me. I have also already done tests only with the GTX 1060, because the memory is larger, no improvement...

Then I tried not to connect a monitor to the GPU with which it is calculated, so that no memory is consumed, no improvement.

What can i do?


Dieter



Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14813
    • View Profile
Re: Not enough memory on GPU
« Reply #5 on: September 09, 2021, 05:17:10 PM »
Hello Dieter,

Have you already checked, if the texturing works on GPU without any CPU fallbacks, if you use tile size = 256 (default)?
Best regards,
Alexey Pasumansky,
Agisoft LLC

Dieter

  • Full Member
  • ***
  • Posts: 198
    • View Profile
Re: Not enough memory on GPU
« Reply #6 on: September 09, 2021, 05:24:18 PM »
I will do that tonight, what is the exact meaning of tile size?

Dieter

Dieter

  • Full Member
  • ***
  • Posts: 198
    • View Profile
Re: Not enough memory on GPU
« Reply #7 on: September 10, 2021, 06:27:34 AM »
Same Problem with tile size = 256

LOG:

Initialized texture renderer
Configuring pipeline...
Initialized memory broker
Configuring mosaic pipeline with outliers filtering
Constructed pipeline
Relaxed precision enabled
Collecting memory requests...
Allocating memory requests...
Estimated required video memory: 4238 MB
Estimated device memory: total 4043 MB, used 67 MB, available 3437 MB
*Cannot use GPU. Reason: Not enough memory on device
Performing blending on CPU...

rendering 1761 pages
*calculating mesh connectivity... ***done in 3.13 sec
detecting outliers... ********
final blending/outliers done. (in 393.874 sec)
Avg camera fetch time: 0.582111 sec
Total stalled time: 0.157 sec
Avg camera processing time: 0.792101 sec
Processed cameras
**postprocessing atlas... ******************************************done in 171.759 sec
applying textures... done in 0.057 sec
selected 0 cameras in 0 sec
selected 0 blocks in 0 sec
*****selected 775 cameras in 30.349 sec
*selected 12275 blocks in 20.331 sec
loaded model blocks in 2.7 sec
loaded uv blocks in 2.271 sec
initializing renderer... done in 0.972 sec
All cameras are fine
Activating context...
calculating mesh connectivity... *done in 3.475 sec
***rendering 0 to 1837 pages
Initialized texture renderer
Configuring pipeline...
Initialized memory broker
Configuring mosaic pipeline with outliers filtering
Constructed pipeline
Relaxed precision enabled
Collecting memory requests...
Allocating memory requests...
Estimated required video memory: 4318 MB
Estimated device memory: total 4043 MB, used 67 MB, available 3437 MB
Cannot use GPU. Reason: Not enough memory on device
Performing blending on CPU...

********************************************************************************************************************************************************************

Dieter

  • Full Member
  • ***
  • Posts: 198
    • View Profile
Re: Not enough memory on GPU
« Reply #8 on: September 18, 2021, 11:24:53 AM »
I have now bought an RTX3060 with 12GB of RAM and replaced the GTX 970, now everything is running as it should.
Obviously, Metashape incorrectly calculates the memory of the GTX970 with 4 GB of RAM.

Dieter