Huge GIS dataset (10000 images, 36MP) SSD and RAM requirements

[ferdywallinx]

Hi All,

I've bought a computer with 32GB RAM and 2TB SSD and am fully aware that this will not do for my datasets. I work with 5000-10000 16-BIT 36MP images, which amounts to 1.5-2.5TB per dataset. I am going to upgrade my RAM but at the same time need to increase my ROM.

I am doubting between 128GB or 256GB RAM and would like to know the added value (decrease of processing time).

Also, since swapping will be necessary with these kinds of datasets, should I buy 2-3 2TB SSD's (540MB/s read/write) and put them in RAID 0 or should i go with one 2TB SSD with 2700/1800MB/s read/write?

In any case, should I process in chunks anyway?

Thanks so much for your help!

[stihl]

Absolutely process it in chunks.
Align datasets of 2000 images or so on high alignment. Then after you've sorted out the GCP's for these areas just merge the chunks. It's quite easy and much faster.

[ferdywallinx]

great, will do! any thoughts about the hardware?

[stihl]

Upgrading to more RAM is always handy. I use four workstations, three with 64GB and one with 128GB. The large amount of RAM has helped me out in numerous occassions so if you have the chance to upgrade to 128GB you should.

Mind you that using more than 64GB of ram requires a Xeon CPU and adequate motherboard.

[ferdywallinx]

Hi Stihl,

I was aware of the Xeon thing so this is no problem.

[Artisan S]

Hmzzzzz,

Why on earth use a Nikon D810 for Agisoft, and shoot 10.000 images....that looks like a gross overshoot to me. What on earth (litterally this time) are you scanning?

But with a set like that I would start to worry about the maximum RAM requirements of Windows 10. You might get a pointcloud at 128 Gb but I sincerely doubt it, sir. At least devide and conquer will be in order.

By the way, buy all the SSD's you like, Agisoft is not in the habit of using SWAP memory.....at least that's my experience (and I have a comparable machine).

Greets, Ed.

[ferdywallinx]

Hi Ed,

well we basically take images from a plane of a few thousand hectares at a few centimeters resolution and then try to stitch them together. I could split this in parts but I am worried that i will then get alignment issues if i do this in separate projects.  I was actually planning to use the SSD's as swap indeed, do you reckon that photoscan does not use swap space and will just crash when overloading the memory?

I guess that if i make chunks it will only load all the data in the memory of one chunk at the time (hopefully). FYI I am not working in windows 10 but linux XFCE distro. Not sure if that would make any difference.

greets,
Ferdy

[Artisan S]

It does not crash......it's polite Russian software....it will inform you that you can save your project if you want to, but it will stop the calculations. Yeps I looked quite puzzled when that happenend.....but usually I can see the memoryload creeping towards my memory limit (32 Gb) and then I stop Agisoft and start using chuncks and allign then in other software that is a bit less memory hungry like Cloud Compare. Now dense Point Cloud generation isn't the most memory intens step (mesh creating needs a lot more). But looking on this forum I know that my machine (though fast enough) does not contain nearly enough memory to do really big tasks like arreal surface reconstruction on a big scale.

Greets, Ed.

[Artisan S]

I don't think the OS does make all the difference, a freind of mine owns a 64 Gb 2 x Xeon Apple with some K cards of mr. Nvidia.....same results, only a bit later.

Greets, Ed.