The time of the build mesh step is increasing and the progress is only 1%

[Majdoleen Awadallah]

Hi all,
I tried to build a project of 7500 images with total size 600GB cover 1200 hectare using 4 nodes after configured the network, and each node is: 11th Gen Intel® Core™ i7-11800H@2.3GHZ, 32GB Ram, Nividia T600.
When it comes to the build mesh steps, it took around 10 days to build the depth maps, and the time of the build mesh step is increasing and the progress is only 1% since last Thursday, what might be the issue here? Do I really need this time?
P.S: on Agisoft Monitoring it shows that each node is maximum spending only 9GB/32GB from the RAM.

Please advise

[Bzuco]

You are probably using hires photos and in combination with T600 quadro, the time needed will be really high.
T600 has only 1.6TFLOP of performance. One desktop low/mid range RTX 2060super gaming card has 7-9TFLOP.
11800H mobile procesor if it is not cooled enough, can not produce fast enough processing times.

[Majdoleen Awadallah]

Thank you so much for your reply, May you please advice is their a way to cool it, and what is your suggestion in this case?

Also If I buy one desktop with low/mid range RTX 2060super gaming card has 7-9TFLOP, what is the approximate time needed to finish the 3d model for the project above? and what is the required Ram needed?

You help is highly appreciated

[Bzuco]

If all your nodes are notebooks, the only solution is undervolt both CPU and GPU and have room with cold air. But that will be still not enough...for you it would be better to have one PC for 2000€ than 4 weak notebook nodes connected with network.

We do not know what kind of photos format are you using, what is the resolution of photos, ...even then it is hard to tell how long your nodes would take finished the project.
If you will be buying something new, choice graphics card from nvidia RTX 3xxx series, they have even more raw FLOAT performacne than my example RTX2060super.  E.g. RTX 3080 10GB version has 20.3TFLOP for 800€. With 16cores/32threads procesor and 64/128GB of RAM you can finish the project much sooner.

Roughly depth maps calculation could be 3-4 times faster, maybe more...I do not know how fast your network is and how long are your nodes waiting for data...etc.

[Majdoleen Awadallah]

Thank you for your reply, If we buy the  gaming desktop i911900KF, RTX 3080, does this will increase the performance? Is this one will work with my project?

In our case, do you suggest to add the desktop as a node and work with other workstation laptops? to continue the process, or do you suggest to tun the project on the desktop with out the 4 laptops?

May you please tell me how to undervolt both CPU and GPU for the 4 nodes?

[Bzuco]

11900xx is not good choice, because 11xxx generation for PC is technologically designed for 10nm fabrication process, but it was realized on old 14nm fabrication process => higher power consumption and temperatures.
Better choice is 12xxx/13xxx gen of intel CPUs or AMD 5xxx/7xxx procesors.

Quick instructions how to undervolt CPU. Download Intel Extreme tuning utility and there you will find voltage offset option. Start to decreasing it with 0.02steps untill your system freeze. Maybe your notebook will freeze at -0.08V offset. Then you need to go back one or two steps to value, when it will be stable (e.g. -0.06V). After each step you should test the system few minutes(2-4) if it is stable. That utility has build in stability stress test.

Undervolting GPU. Download MSI afterburner utility. Set power limit to 70% and increase core frequency +80-120MHz(this value depends on chip generation and exact model). Maybe you will be not able to change power limit so much or at all...it depends on GPU generation and certain model. I had never quadro card, so it can be locked by manufacturer. You will see.

...this is just quick advise how to undervolt. Better solution is lock GPU/CPU frequencies at some value and set much precise voltage value, not just offseting it.

If you have very fast network you can add PC as a node, but I never run metashape network processing, so I do not know how the data are distributed and how effective it is in comparisson with just one workstation.
In one PC workstation you are limited only by system RAM memory bandwidth and maybe old HDD clasic disk. With modern SSD disk it should be no problem. The rest like CPU and GPU components behave mostly in linear dependence...more CPU cores and more CUDA GPU cores or higher frequencies means linearly more performance....onyl some metashape single thread processing subtasks can disrupt this linear dependance.

[Majdoleen Awadallah]

Thank you so much for your promote help, unfortunately I tried to install the Intel Extreme tuning utility but as written here "But in case of Intel Core i7-11800H and Windows 11 x64 OS it is impossible to start the application at all". (https://community.intel.com/t5/Processors/Intel-Extreme-Tuning-Utility-Intel-Core-i7-11800H-support/td-p/1382592)

Do you have any suggestion?

[Bzuco]

https://www.youtube.com/watch?v=cqVnkEeISUE

Try to download older version of iXTU, that guy in video is running 7.4.0.26 and it is working for him.
Hi is also showing how to undervolt in XTU, in HWiNFO monitoring tool the temperatures and also CPU package power consumption max. ~82W at 4.2GHz...my desktop 11700F procesor is pushing at that frequency 122W after undervolting...that is what I was mentioning(desktop version is on 14nm older process-not worth to buy for desktop, notebook version is on better 10nm.)

Cinebench is just short test. You will be probably able tu run long metashape CPU tasks at all core 3.6GHz(normal FAN speed) or ~3.8GHz(maximum FAN speed). In both cases without hitting 90+°C and without frequency throttling. You can also compare what is the CPU frequency now and then after undervolting.

EDIT: in video comment section somebody is mentioning that on some machines you can undervolt also in notebook bios...if XTU does not work.

[Majdoleen Awadallah]

Thank you so much for your great help, really appreciated!

We bought a new desktop yesterday( the specs are as shown in the images attached) and tried to run a small sample on around 670 images and while building the mesh on high mood, it takes around 6 hours to build the depth maps and gives that it needs around 11 hours to finish the mesh, Is this the normal behaver?

Also we noticed the the GPU is not working, the performance on task manger show the GPU 0% or 3% maximum!
is this normal?

We installed the latest version of the driver, Any suggestion is highly appreciated

Thank you in advance

[Bzuco]

Be sure you have enabled GPU in settings and disabled the bottom option(attachment).
670 depth maps(37Mpix if this is correct info from your screen) on RTX 3090 should be calculated under one hour(my estimation).
Mesh generation process is GPU accelerated only in few subtasks, the rest is dependant only on CPU.

What is your original photos resolution in pixels?

EDIT: I forgot screen in attachment

[Majdoleen Awadallah]

The settings is similar your attached as shown, also you can find the details of each image ( it is an areal photo and not drone),

the depth maps steps is exactly took 3 and half hours!
Please advise.

[Bzuco]

Then it seems 3 hours time for depth maps correct. During this process CPU is also quite high utilized.
Do you have all the project files on SSD disk 0 or on the external disk 1?
How was utilized GPU/CPU and disk read/writes during depth maps calculation? can you show some larger/longer graph from task manager or some other monitoring tools?

In mesh generation process, as I said, it is mostly CPU task, and only some part are GPU accelerated. If I am correct, metashape is generating much denser mesh during generation and then it is decimated, which is in my eyes a bit waste of time.

I am using cloudcompare for generating mesh from metashape pointcloud, where I can set the density of mesh in metric units.

Do you need High quality of depth maps and mesh model? Wouldn't be also Medium quality enough?

[Majdoleen Awadallah]

Hi Again,

I am using SSD 0, unfortunately I don't have graphs while building the depth maps (attached graphs when starting the generation mesh steps).

Please note that I followed the steps mentioned here to reconstruct a 3D model.

Regarding Medium or High, At the end I want w 3D model with good details, What is the main different between high and medium?

[Bzuco]

Depth maps quality:
Ultra High - full photo resolution is used,  14204x10652
High - 1/2 of original res. - 7102x5326
Medium - 1/4 of orig. res. - 3551x2663
...
...
From you photos you can estimate(or quite preciselly calculate) how big area covers one pixel in reality...e.g. if 1pix is  3x3cm, your maximum precision is 3cm. If 10cm precision would be enough for you, then you can set High(6cm precision) or Medium(12cm precision) quality.

Build mesh:
Face count - you can't really tell if high/med/low or custom value will be enough until you see the final result. So it is trial/error task. City area would need more polys, and flat areas less....every project is different.
Try create mesh on some smaller project and you will be able guess better if e.g. 500k polygons is enought for some object/area. Then try to guess what polygon count will be enough for your big project. The bad is, that metashape is still creating much denser model, which is in the last step decimated to your desired polycount.

I am often using just point cloud for presentation( https://github.com/potree/potree ), because size of points can be adjusted and I still have option to measure everything.

[Majdoleen Awadallah]

Thank you for your clarification, I want a 3cm resolution, and I took you suggestion, I tried it on a small area and the high depth map does not give me the quality required, also their is some detorsion in the building, what is the main reason for this?

Also I have this question, My final result is slpk file, If I skip the build mesh and texture steps and go to the tiled model step (you can export the model to slpk formal direct), does this affect the resolution?

What is the main different between mesh and tiled?