Project with 220K images for processing

[jivago]

Hi
I have a huge project about 170 Sq. Km and wanna fly with Phantom 4 RTK.

Base on my calculation I will have about 220K images from 100 m height above ground with overlapping 75% in both in both directions.

Does anybody know anything about best practice for processing such a numerous amount of images?

sincerely,

[JRM]

My advice ? Don't use that kind of drone for such a big area, a fixed wing drone will be done in a fraction of the time (depending of the legislation in your country for out of sight flight).

All the problems you will encounter with this amount of photos in metashape is nothing compared to the nightmare that will be the battery management of the P4.

[jivago]

I'm agree with you but I don't have a fixed wing. If I shall handle battety management, is there any advice for doing processing in ideal situation?

[wizprod]

Maybe consider the cloud processing option, that Agisoft provides?
You might also look into dividing the area into chunks.

However, I also agree that the processing part isn't your biggest issue, the flighttime is. You would have move location many many times, and change battery for well, probably every 1 sqkm (probably quite optimistic), making it 170 changes. A small calculation with 16 minutes of flight time makes for ~45 effective hours in the air. That could almost pay for a fixed wing setup.

[Alexey Pasumansky]

Hello jivago,

I think, for alignment it is better to split the area into four blocks up to 60 000 images each and process them separately.

Prior to the processing make sure that the altitude information is correct in each block (chunk) - all altitude values for each flight inside every block are measure from ellipsoid (or from geoid, if compound coordinate system is selected). Otherwise, if each flight has its own base level for altitude, you might face alignment issues and artifacts in the flight overlap areas.

[Corensia]

What kind of computer specs would you need to run a 60,000 image project? Has there been internal testing of projects of that size?

[jivago]

My map scale is 1:500 therefore I think very rare fixed wing can fly for that purpose.

I convinced myself to fly about one month and have no problem with that but I don't know which pc with what configuration can processing more than 20,000 images in each chunk!

My record till now is 20,000 images. Does anybody have suggestions? What's is your opinion about HP Z840 workstation?

[SAV]

Hi jivago,

That's a huge area.
I highly recommend to increase your AGL to be able to cover a larger area with a single battery. Depending on your local drone legislation, you can apply for an exemption for the AGL limit. For example, increasing AGL to 300m would still give you a GSD of 8.2cm. I have recently done a larger scale survey over an area of 18km2 at AGL 300 with 80% forward and 75% sideward image overlap. I was able to safely cover 0.75 - 0.80 km2 per battery. I wish the Phantom 4 Pro/RTK would ascend and descend a little quicker. It takes quite a long time to get up to and down from AGL 300m. However, AGL 300 comes with the advantage that it allows you to fly much faster (I was doing 13m/s) without getting into motion blur issues in your imagery. Blurry pictures need to be avoided because the lead to higher error/lower accuracy.

And as suggested by Alexey, you'll probably need to split your project into a few smaller chunks for processing.

Regarding hardware, this might be some useful information for you:
https://www.pugetsystems.com/recommended/Recommended-Systems-for-Agisoft-Metashape-145/Hardware-Recommendations


All the best. And it would be great if you could share how you finally did this massive survey.

Regards,
SAV

[jansawicki]

Hello jivago,


Our company provides services for processing large amounts of image data  sets on a high performance workstation. If you are interested in our help, please direct message me - we will find a way to cooparate

Cheers

[jivago]

Hi jivago,

That's a huge area.
I highly recommend to increase your AGL to be able to cover a larger area with a single battery. Depending on your local drone legislation, you can apply for an exemption for the AGL limit. For example, increasing AGL to 300m would still give you a GSD of 8.2cm. I have recently done a larger scale survey over an area of 18km2 at AGL 300 with 80% forward and 75% sideward image overlap. I was able to safely cover 0.75 - 0.80 km2 per battery. I wish the Phantom 4 Pro/RTK would ascend and descend a little quicker. It takes quite a long time to get up to and down from AGL 300m. However, AGL 300 comes with the advantage that it allows you to fly much faster (I was doing 13m/s) without getting into motion blur issues in your imagery. Blurry pictures need to be avoided because the lead to higher error/lower accuracy.

And as suggested by Alexey, you'll probably need to split your project into a few smaller chunks for processing.

Regarding hardware, this might be some useful information for you:
https://www.pugetsystems.com/recommended/Recommended-Systems-for-Agisoft-Metashape-145/Hardware-Recommendations


All the best. And it would be great if you could share how you finally did this massive survey.

Regards,
SAV

Hi SAV

thank your for your advises, but I have to say my map scale is 1:500 and need image resolution below 5 cm based on local instruction.

There for I have to fly at maximum height about 120 meter and for avoiding image motion blur issue restrict drone speed at 9 or 10 m/s.

And for hardware configuration I think below specification is enough for my project and splitting project into smaller parts is inevitable.

Cpu : 2x Xeon E5 2688 V4
40 Core 80 Thread 100 MB Cache

MB : Asus Z10PE-D16

Ram : 64GB ECC DDR4 2400

SSD : SAMSUNG 970 EVO M.2 2280 1TB
Hard : WD 4TB Red

VGA : AORUS GeForce GTX 1080 Ti Waterforce Xtreme Edition 11G Rev 1.0/1.1

Case : Mastertech T500
Power : Green 1050 OCPT
2xWaterCooler Green 120 Evo

regards,

[ruyi7952]

220k？ 22w? In China, this is a normal project.

And the way we do it is very simple,break up the whole into parts，If the geometric strength and precision of each control point are sufficient, the quality of the whole project is very good.

If possible, you can entrust us 

[jivago]

220k？ 22w? In China, this is a normal project.

And the way we do it is very simple,break up the whole into parts，If the geometric strength and precision of each control point are sufficient, the quality of the whole project is very good.

If possible, you can entrust us 

We are gonna do it in Iran and aside from whatever processing unit, I have to break project into smaller parts.

[SAV]

Hi again jivago,

Looks like a powerful system. But I would not go for a Xeon processor, based on a statement by Pugetsystems:

Xeon Scalable processors are not worthwhile for PhotoScan

Here are the details: https://www.pugetsystems.com/labs/articles/Agisoft-PhotoScan-1-4-1---Xeon-Scalable-CPU-Performance-and-GPU-Scaling-1164/

I would only go for a Xeon based system if I wanted/needed more than 128GB RAM.
Intel's Core X an Core i9 series are much better value for money. I personally would go for the i9-9900K. It's 'only' a 8 core processor, but it outperforms a 18 Core 9980XE, as shown here: https://www.pugetsystems.com/labs/articles/Metashape-1-5-1-Intel-Core-i7-Core-i9-vs-AMD-Threadripper-Performance-1465/

So don't get hooked by the number of cores only. That's not the only thing that controls your processing times.

Regarding your GPU choice. I suggest to use the money that you save by not using a Xeon setup and invest it in the latest RTX 2080 Ti 11GB or even a dual RTX 2080 Ti 11GB setup. You might even have money left to upgrade to 128GB RAM as well.


All the best.

Regards,
SAV

[jivago]

Thank you very much SAV, I reached out your suggestion after searching and completely agree with you. I decided to use more than one Graphic card and select a CPU with higher clock and usual number of cores instead of Xeon series.

regards,

[SAV]

Great to hear 

All the best for your massive project.

Regards,
SAV