Forum

Author Topic: Metashape RAM usage  (Read 10432 times)

GrinGEO

  • Sr. Member
  • ****
  • Posts: 262
    • View Profile
Metashape RAM usage
« on: January 31, 2019, 05:03:32 PM »
Hello to all
On the website comparition between Agisoft & Metashape it is said that using depht maps the memory usage has been reduced a lot. Unfotunately some projects still crash due to Memory overload in Building Mesh process, almost at the end.

Would it be possible to update this table?
https://www.agisoft.com/pdf/tips_and_tricks/PhotoScan_Memory_Requirements.pdf

Becouse it would help a lot having some better advices how RAM we need to install on the systems. Maybe the calculation could be done using 20 MPixel images (wich is almost standard today for many applications)

Thanks a lot

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14816
    • View Profile
Re: Metashape RAM usage
« Reply #1 on: January 31, 2019, 06:06:22 PM »
Hello GrinGEO,

Can you send the processing logs related to the failed operations and the project properties (for example, as PDF report file) to support@agisoft.com, so that we can understand, at which point of the processing the issue occurred?
Best regards,
Alexey Pasumansky,
Agisoft LLC

GrinGEO

  • Sr. Member
  • ****
  • Posts: 262
    • View Profile
Re: Metashape RAM usage
« Reply #2 on: January 31, 2019, 06:12:58 PM »
yes, I need to restart calculation first.

But is it possible o uplaod that datasheed so that we understand better how much RAM is necessar? Becouse I want also to upgrade workstation and not shure if we still need 128GB RAM or if 64 are enough

Mak11

  • Sr. Member
  • ****
  • Posts: 374
    • View Profile
Re: Metashape RAM usage
« Reply #3 on: January 31, 2019, 06:21:00 PM »
GrinGEO

The log would help to figure out if the crash happened because lack of RAM or because the model was too dense (100+Millions of polygons) resulting in the crash of the GPU because it couldn't render it. Like here:
https://www.agisoft.com/forum/index.php?topic=10316.0

Cheers

Mak

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14816
    • View Profile
Re: Metashape RAM usage
« Reply #4 on: January 31, 2019, 06:40:15 PM »
Hello GrinGEO,

If you wish, you can send the original image set and the project with the alignment results, so that we could run it on our side and check the memory consumption.
Best regards,
Alexey Pasumansky,
Agisoft LLC

GrinGEO

  • Sr. Member
  • ****
  • Posts: 262
    • View Profile
Re: Metashape RAM usage
« Reply #5 on: January 31, 2019, 09:32:28 PM »
I sent you the log created by my mobile workstation with 16MB Ram.

But Im wondering... instead of creating the Shapes from Dense Cloud or Depth Maps, why not create from DEM? Would that not be much more easy and less power consuming?

Becouse on Depth Maps for example I even don't know what the results will be... while creating from Dense Cloud it gives me an aprox. calculation of the result. It is a very pain that Metashape crash almost at the end of the calculation and that there is no save in between.

btw. how can the Depth Maps be viewed? I see them in my chunk but I can't open it.

In this 3D Survey Video it looks so simple to create mesh & do calculations... I think if we can improve this in Metashape, this would be very great.

https://www.youtube.com/watch?v=PsT-OMXj8w8

GrinGEO

  • Sr. Member
  • ****
  • Posts: 262
    • View Profile
Re: Metashape RAM usage
« Reply #6 on: February 01, 2019, 11:27:23 AM »
Still crashes on my systems with 16GB RAM for a 1.8M Mesh creation. Im wondering if virtual memory from an SSD can help in this? I understand it is slower but would this help to do the entire calculations? Unfortunately on my old Workstation I can't upgrade memory

GrinGEO

  • Sr. Member
  • ****
  • Posts: 262
    • View Profile
Re: Metashape RAM usage
« Reply #7 on: February 01, 2019, 02:23:41 PM »
Alexey, m wondering about something and maybe you can explain this better as an expert.

I have aligned 280 images in highest mode, created a Dense Cloud with about 82M points. Now Im creating the Mesh from Dense Cloud (Not Depth Map that should use less memory) in High Mode.

From the launch windows for Mesh calculation I got an indication that I should receive about 5.5M faces, wich is pretty high becouse normaly I do only less than 2M but I want to try if with virtual memory Agisoft can handle this high caluclations.

From the screenshot  you'll see that Metashape is using almost 40 GB of Memory (16 GB modules installed, 24 GB are virtual on SSD) an since yet MS didn't crashed.

But if you see the Detail Log it looks like that MS first created a Mesh with over 258M faces and is now decimating. Why does MS calculate such high mesh? It looks like to me that MS first meshes every Dense Cloud point with at least 3 other points to create the first Mesh Step.

But would it be not a much faster process that MS, when selceting High, Mid, Low Mesh, skips ponts in betwen and creates mesh from every second, third or fourth point? Something similar like exporting the DEM with a certain Raster.

I think this would reduce a lot the memory and CPU load and improve a lot the workflow.

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14816
    • View Profile
Re: Metashape RAM usage
« Reply #8 on: February 01, 2019, 03:45:19 PM »
Hello GrinGEO,

If you are using Arbitrary mode for the mesh based from the dense cloud, the target number of polygons doesn't matter, at first the mesh with the max possible faces will be generated and after that it will be decimated to the target limit. So here the memory consumption depends mostly on the number of points in the source dense cloud and scene complexity.

There's no direct connection between number of points in the cloud and the number of polygons in the max resolution mesh, as it depends on the point distribution.

Currently we do not provide tools for the dense cloud decimation / filtering, however, you can export the dense cloud to the external application, reduce the number of points and import it back to perform the meshing based on that dense cloud.

Also depending on your needs, you can consider using Height Field meshing option to get 2.5D model - it will require considerably less RAM compared to Arbitrary method.
Best regards,
Alexey Pasumansky,
Agisoft LLC

GrinGEO

  • Sr. Member
  • ****
  • Posts: 262
    • View Profile
Re: Metashape RAM usage
« Reply #9 on: February 01, 2019, 04:45:27 PM »
Yes alexey I know that Height Field is faster but for visible 3D models for clients the Height Field looks very ugly when Trees are involved. If we could improve this I would be very happy.

GrinGEO

  • Sr. Member
  • ****
  • Posts: 262
    • View Profile
Re: Metashape RAM usage
« Reply #10 on: February 02, 2019, 03:18:29 PM »
USING VIRTUAL MEMORY RESOLVED A LOT OF MEMORY PROBLEMS on my Computers

I was really unhappy about Metashape crashing so a lot, and I completely forgot virtual memory as a feature becouse today, using SSDs, all our computers have deactivated it. I will still upgrade our mobile workstation to 32 GB memory (purchased it today) but with virtual memory assigned in Window you can achieve a lot of more Mesh calculations than bevore, how did I resolve it.

I added:

C: SSD 16.384 MB virtual memory
F: SSD 16.384 MB virtual memory
V: SSHD 32.768 MB virtual memory (but I have seen that Agisoft didn't used it)

I know doing virtual memory on an SSD is not very good for the SSD, since it reduces a lot the lifetime due to the high write reading sequenses, but since on each computer we have a 200 GB SSD MSATA for system and a 1 TB SSD SATA for project files, I used that as virtual memory upgrading my PC from 16 GB (wich caused a lot of out of memory in Agisoft) to 81.920 MB total available to the sistem.

A project with about 250x 12 MPixel images took about 40 GB of total memory space to calculate mesh in high-mode from dense cloud (high quality 82M points)
So a 500x image project should need around 80 GB of memory to do the same calculations.

Unfortunately, rendering mesh from Dense Cloud is not only memory consuming, but also very high CPU consuming and it took something about 4 days, becouse first it created a super high 282M mesh that was reduced to about 5M later. This is, in my opinion, a not very efficient workflow resolved yet in Agiosft and I hope they can optimize this.

So i switched to Depth Maps mode and it took in high & high mode about 4 hours on the sistem creating a 4.7M shape, wich is almost what I planned in doing in dense cloud mode.

But also in Depth Map mode my computer crashed bevore, including virtual memory resolved those problems of OUT OF MEMORY.

I hope this might help someone else here in the forum.

At moment Im going to plan to purchase a Threadripper sistem, but instead of buying other 64 GB of DDR4 RAM to have a total of 128 GB, I will spend those moneys in a second GPU.