Hardware for Large Scale Projects

[UsmanJam]

Hi All,

We are presently engaged in the implementation of an extensive project encompassing 180,000 images captured at a resolution of 45 megapixels each. The project involves the generation of Orthomosaics, Digital Surface Models (DSMs), Point Clouds, and Mesh Models.

Would appreciate it if you can recommend the right hardware configuration. Intel Xeon? i9? RTX 4090? single workstation or multiple? 2x Xeons? 2x RTXs? whatever it might be. Cost of the hardware is not the issue here.

Best Regards,

[Bzuco]

With 180k photos the key will be good utilization of HW. I see the fastest way by using network processing locally on single machine(maybe two).
If you create several local instances of running metashape, you can e.g. greatly speed up phase of detection points from photos, which is important in case of 180000 photos.
These local instances are helping to keep CPU and GPU utilized ~100% all the time without drops.

Instead of buying AMD Ryzen Threadripper PRO 7995WX you can spend much better money and buy two computers with 32c64t Threadripper 7975WX(higher multicore frequency) and RTX4090.

What will be important during processing is manually assigning the total number of nodes in each processing phase.
For example:
First phase - detecting points needs maybe 10-20 concurently running metashape instances to feed enough one RTX4090...so you enable 10-20 nodes.
After this pahse, matching points is more excesive task for GPU, so you will keep only ~3 nodes active, or maybe few more if you see GPU is not utilized without % drops. More instances would feed too much GPU.
During depth maps generation, which is pure GPU task, maybe two GPUs would be useful.

So my final advise is, buy one computer and if it will be fully utilized locally during network processing, then you can buy exactly another one to half the processing time...if needed. Or you can buy second GPU if you will see, that depth maps gen. is still slow.

It could be beneficial to buy two RTX4080 instead of one 4090, from performance reason, but that is only my assumption.

[UsmanJam]

Thank you Bzuco for the help.

Do you think going for a Intel Xeon is a better choice or the i9 24c? Just wnated to check for the CPU as well, which one would be the best?

Thanks again!

[Paulo]

With 180k photos the key will be good utilization of HW. I see the fastest way by using network processing locally on single machine(maybe two).
If you create several local instances of running metashape, you can e.g. greatly speed up phase of detection points from photos, which is important in case of 180000 photos.
These local instances are helping to keep CPU and GPU utilized ~100% all the time without drops.

Instead of buying AMD Ryzen Threadripper PRO 7995WX you can spend much better money and buy two computers with 32c64t Threadripper 7975WX(higher multicore frequency) and RTX4090.

What will be important during processing is manually assigning the total number of nodes in each processing phase.
For example:
First phase - detecting points needs maybe 10-20 concurently running metashape instances to feed enough one RTX4090...so you enable 10-20 nodes.
After this pahse, matching points is more excesive task for GPU, so you will keep only ~3 nodes active, or maybe few more if you see GPU is not utilized without % drops. More instances would feed too much GPU.
During depth maps generation, which is pure GPU task, maybe two GPUs would be useful.

So my final advise is, buy one computer and if it will be fully utilized locally during network processing, then you can buy exactly another one to half the processing time...if needed. Or you can buy second GPU if you will see, that depth maps gen. is still slow.

It could be beneficial to buy two RTX4080 instead of one 4090, from performance reason, but that is only my assumption.

Hello Bzuco,

your suggestion is very interesting. As I am not a computer expert, can you explain how you create several local nodes of Metashape to work in network... Would be very helpful to have a step by step guide for such setup.

Thanks again for your insights on computer use!

[Bzuco]

@UsmanJam
Intel Xeon are on older manufacturing process, so AMD Threadrippers are slightly better options. I would stay with Ryzen 7950x(16c32t), but yeah, if monney are not problem, then threadrippers another 16 cores are very useful .
Intel i9 24c32t is just (8 performance cores/16t) + 16 efficient slower cores without additional threads, so it is only good choice, if price/performance and performance/watt is better then AMD CPU...so in your case not good options at all.

@Paulo
Here is guide for network processing locally I found on YT. https://www.youtube.com/watch?v=BYRIC-qkZJ8
Also check Metashape PRO PDF manual for the latest command line syntax, in the video is used a little bit older.
For running more instances on one PC just, run node.bat more times  and then you can pause and resume workers in build-in monitor application according your needs at every stage of processing.

[steve3d]

Well, I'm currently working on an project which have 35k photos, and 40M pixels of each photo.

I'm using three workstation and an NAS to store the project. From what I've already learned, you should at least have 7 times larger space of the photo size, which means if your photo is 1T on disk, then you should at least have 7T spare disk space. And of course, You can not use HDD or any HDD RAID, that's extreme slow. I only used 3 workstation, But I've observed 7Gb network bandwidth at some moment.

And for the generation, I can say, you'll need some nvidia cards which have at least 24G VRAM, or you might encounter this problem: when generating textures from blocked model, there is not enough VRAM to calculate on GPU, then you will end up with calculate on CPU, which is at least 20 times slower.

And for model generation, metashape have some defects. When you creating the blocked model, which is a must have option for this kind of large dataset, at least 50% of the calculation work is single thread, only need single core. So you'll need a high freq CPU and of course, high freq of memory.

And when generating the textures of blocked model, there is some improvements needs to be done in next release? The problem is, if you have 300 blocks of model, then when generating the texture, each block need to read all 300 blocks, which takes at least 80% of time. and for the last step of generating the texture of blocked model, you'll need a lot more more more memory, otherwise, It will simply fail because there is not enough memory.

My experience is based on only one chunk of 35k/40M pixels dataset. Because I need to modify the generated model, without the blocked model, I think it's impossible to modify the mesh.

Of course, If you need only the tiled model, That's totally another story, just avoid the high polygon count setting. I've tested with high setting, my dataset gives me 95G cesium tilesets. 

[olihar]

This is an absolutely Threadripper cluster solution that needs to be created here, given the kind of data has been and will be shot in the future.

[UsmanJam]

Hi Bzuco,

Thank you for the very useful information.

Just one question out of curiosity, is there a reason why the AMD Ryzen Threadripper PRO 7995WX (96 Cores | 192 Threads | 5.1GHz Boost) is not a good choice? Is it purely due to the cost of the product?

[olihar]

Hi Bzuco,

Thank you for the very useful information.

Just one question out of curiosity, is there a reason why the AMD Ryzen Threadripper PRO 7995WX (96 Cores | 192 Threads | 5.1GHz Boost) is not a good choice? Is it purely due to the cost of the product?

7995WX makes no sense for this.

[Bzuco]

Hi Bzuco,

Thank you for the very useful information.

Just one question out of curiosity, is there a reason why the AMD Ryzen Threadripper PRO 7995WX (96 Cores | 192 Threads | 5.1GHz Boost) is not a good choice? Is it purely due to the cost of the product?

The price is of course one of the reason, but not the only one.

1. all those 79x5WX threadrippers(16/24/32/64/96 cores variants) have thermal design and ~power limit set to 350W. If you don't touch anything in terms of decreasing power consumption and make CPU more efficient(undervolting processor), then in case 7995WX you will really stuck on base frequency, which is just 2.5GHz...in case all your cores will be utilized.
CPUs have also 3 levels of internal caches which speeds are dependent on CPU core speed. Cache memory bandwidth is lower if core clock is just 2.5GHz than e.g. @4.0GHz... from that reason is better spent those 350W on less cores with higher clock speed.

2. I am not sure if all those 96 cores would be utilized enough, because you will be still limited with RAM memory bandwidth.

3. at the cost of one 96c CPU you can buy two PCs with two GPUs, which will give you more performance.

[loggie24]

We have a workstation with a 7995WX, and from my experience the main benefit of it would be if you are running multiple Metashape processes at once, that way you can actually use most of your cores, otherwise with a single process it actively uses maybe 20 or so which certainly isn't optimal.

[SimonBrown]

I'm working on a prerequisite that each of the 180,000 images carry optimal overlap with their neighbours?

If yes then its a hardware discussion.

If not, then excessive overlap is a known issue that will dramatically increase processing times. One real world example of how a different approach was taken to solve the processing times issue:

https://accupixel.co.uk/2022/05/13/photogrammetry-consultancy/

No one - particularly those who work in a less structured environment where mission planning with preset overlap variables - will want to leave site with "the one" frame missing. This can be a common issue when working underwater.

It can be better to cull the images after capture and only process what is necessary.

To reiterste - I do not know the full details and it may not be applicable here.