Strange workstations testing results.

[Triplegangers]

Hey Milos! Thank you for sharing that info!

I second that! Switching off Hyper Threading (HT), did show nice improvement in processing speed. However switching off Turbo Boost only crippled general productivity in my system. As my Xeon's boosting up not just for a second or two, but do work over 3GHz during the whole process. May be its due to cooling system. So I can confidently say boosting helps a great deal.



So funny thing about HT. Apps either support it, or they don't. Never knew it can work against you. Its like if all of those employees, (in Wishgranter post) starting to argue with each and screw around pissed off

Also my tests showed that HT OFF only helps during Photo Align process. Geometry Build showed the same time with it being ON or OFF.

IMHO, based on the testing data, Photoscan doesn't support HT, and has a faulty algorithm(s) responsible for Photo Align process. Sounds reasonable to make it not use HP feature, unless we benefit from it.

Would be nice to hear what Agi Team has to say.

[Infinite]

Hey Milos! Thank you for sharing that info!

Would be nice to hear what Agi Team has to say.

Agreed! as this could have possible impact on how future users build rack server for 'possible' network processing!! 

[RalfH]

Yet another test. I have used the sample01_align.psz from the test data set (http://downloads.agisoft.ru/photoscan/sample01.zip) and tested building geometry with 1, 2, 3... 8 cores of an 8-core Xeon workstation (2 quadcore CPUs), processing with CPU only, no hypertheading.

The results are very interesting as they show that multicore efficiency is very low. Using all 8 cores you get only 2.5 times as many samples per second as you would get when using a single core - 69% of the total computating power are lost to "management" and synchronisation. Computation speed per core drops from 18.2540 Mio. samples per second (one core) to 5.7956 Mio. samples per second (8 cores).

Running two parallel instances of Photoscan seems to be a possible solution to reduce total processing time (if you have several projects to be processed), because the total number of samples per second is higher for running two projects at the same time than running them one after the other. Assigning a separate group of cores to each Photoscan instance is slightly more efficient than letting them share all 8 cores on their own terms. Assigning cores by physical CPU further slightly increases efficiency.

Of course this sheds an interesting light on batch processing where several tasks are processed one after the other. Instead of than processing several chunks in one project using batch processing, it would be much more efficicent to split the project and process the different chunks in separate parallel instances of Photocan.

[Wishgranter]

Ralf, for depht stage are here GPUs, up to 10-15 times faster per GPU, depht map is prepared to be processed on GPU stage not CPU, because it could teoretically be possible make LITTLE faster but it would need a lot of effort do it... CPUs are not ideal for data like this, they are great at "serial" work, GPUs are best for paralel work, because we have say 1500+ cores for disposal, on CPU we get 8 cores ( its not so easy to compare but as a guideline for understanding ).......

For now Pscan is slow on ALIGN and MESH stage, im think later can AGISOFT team do it on GPU but its not a so easy task, if it could be ported to GPUs only then we will not need even network ready solution ( what is even more problematic to develop ) so with Align on GPUs we could see inprovement in aprox 10-20times from what im know about the ALIGN stage, with MESH generation even more....... why ? because GPUs are ideal for task like this.... 

with ONE 7970 im get aprox 690 Mil. samples.... so its a 38x faster...

[Diego]

IMHO, based on the testing data, Photoscan doesn't support HT, and has a faulty algorithm(s) responsible for Photo Align process. Sounds reasonable to make it not use HP feature, unless we benefit from it.

I think it's wrong appreciation. For my tests are totally contrary.

HT On Much faster. Maybe it has to do with Board, Memory or Operating System, do not know.

My results with the proposed example.

Machine 01

Dual Intel Xeon X5677 @3.47 GHz. (4 Cores CPU; 4 Cores Virtual HT)

HT On 16 Cores Finished processing in 118.002 sec

HT Off 8 Cores Finished processing in 145.382 sec

Machine 02

Intel Core i7 3740 QM @2.70 GHz. (4 Cores CPU; 4 Cores Virtual HT)

HT On 8 Cores Finished processing in 140.927 sec

HT Off 4 Cores Finished processing in 180.951 sec

Clearly the HT works very well, and if Agisoft algorithm takes advantage, at least in my case, with two machines, of very different Hadware.

[Wishgranter]

you forget to mention what OS you using !!!! as MOST importaint thing.... all up to date with updates or

[Diego]

Hi Wishgranter,

Windows 8 Enterprise x64, Updates On, all time

[Wishgranter]

hmm this is really very interesting !!!! so we will need do more test.......

And dont be suprised, but Xeons and i7 are the same hw inside, only the cache size is different and some microcode is different, architectural its the same thing.....

[Diego]

I agree

[Wishgranter]

hmm we use W2012 Datacenter edition with few tweaks ( me and Alexander ) so im wil dig intro this, because of my digging on what OS to use im stuck with this, as im get here better results on Nvidia GPU. the Datacenter edition should be the fastest one ( long story ) from all WIN versions.......
But as im see now, then WinShitt have done great work to criple it so that people will need upgrade hw and sw..... its hidden, so not everyone get it easy way out......

   

[Alexey Pasumansky]

Hello all,

Thank you for detailed experiment description. They are quite informative and interesting.

We consider that there could be some saturation threshold of core number when the control of processing and task paralellizing requires much resources and they can no longer be used for calculations itself. Even the fall of performance could be observed.

And as Wishgranter has already mentioned under different OS the multithread performance could differ.

[James]

Hey James, from what I know caching is not something that happens for no reason. You have to really sweat a little on the code side to make caching possible.

I would say I wish I'd paid attention in operating systems and computer systems architecture lectures, but I did pay attention and it didn't help...

[RalfH]

Wishgranter,

yes, I know that GPU would be much faster for depth reconstruction, but the machine I'm working on doesn't have a powerful GPU, so CPU is all I can work with for the time being. I think that my test results can be useful for everybody who has to work with a multi-core CPU only.

Anyway, I have done the same test series for the Photoscan align stage (same data set as before), and the results are partially similar but not quite as bad as for depth reconstruction: Total core time (seconds x number of cores) for the task increases as more cores are used. If multi-core efficiency was 100%, total core time would remain constant. Using all 8 instead of only 1 core almost doubles total core time - 47% of the total processing power are used for "management" and synchronisation.

Interestingly, processing of two parallel align tasks is only slightly more efficient as running them one after the other, and core or CPU sharing brings counterintuitive results (with the total core time increasing instead of decreasing if a separate group of cores is assigned to each Photoscan instance).

P.S.: Forgot to mention that this and the previous test were run under Windows Vista 64 bit, using Photoscan standard edition, version 0.9.1 bulid 1621.

[gEEvEE]

Hi all,

for what it is worth, I have reported this problem in a private email already three months ago to Dmitry and Alexey. I did not receive any answer so far, but will just share that email and included results.

I am now back from a National Geographic campaign in Greece, where we documented the Bronze Age site of Akrotiri. I returned with thousands of photographs and during the processing of them, I noticed some very strange CPU behavior in PhotoScan (the latest version, build 1640). What I will describe was only noticed now, so I do not know if it was also an issue in previous versions.

I ran several tests on my two computers (both running window 7 ultimate 64-bit) that I have. In my home PC, I have an Intel Core i7 Extreme 980X 3.33 GHz with 6 cores (so a total of 12 threads) and 24 GB of RAM. On our workstation at the institute, we have two Intel Xeons E5-2690 2.9 GHz, good for 2 * 8 cores  or a total of 32 threads and 128 GB of RAM. This Xeon processer is so far the third fastest ever (http://www.cpubenchmark.net/high_end_cpus.html) and thus a real processing beast.

However, photoset alignment on my home PC is at least twice as fast as on our workstation. This is very annoying, since this super PC has cost us € 10 000 and was bought just for PhotoScan.
I aligned about 2200 photographs on my home PC and on the workstation, but “looking for image pairs” was 2.5 times slower on the workstation (although all processors worked 100 %).

Afterwards, I did a test with 50 JPEGs: these were the results: see comparison.png

As you can see, finding interest points is slightly faster on the workstation, but the pair selection is at least twice as slow. Can it be that this is not at all optimized for two physical processors and that the computation is done double?

To make sure that the CPUs and the motherboard and not wrongly configured, I ran some benchmarks (also of the GPU, but both have an Nvidia GeForce GTX 580 and this is not of interest for this SfM step). The results below show several highly intensive computations performed with the Performance test of Passmark (http://www.Passmark.com). AN overview of the CPU test can be found here: https://www.dropbox.com/s/l6suugfgonxx66z/CPUs.jpg

As you can see, the workstation outperforms the home PC in ALL possible CPU computations.

What could be the cause that PhotoScan performs so much slower on the super PC, although both PCs have all processers 100% in use?
I really hope that you can help us out with this.

Best regards,

Geert

[Wishgranter]

GeeVee, what OS you used for tests ?? because it seems that its something broken in Windows, not a hw issue, but much more a sw issue...... Have prepared my hdd for instalation of  Win XP, 7, 8, Win Server editions, Ubuntu and OSX, im will try test it on every mentioned OS and afther that we can get some data that can show what we canot see..... from my perspective its not a pscan fault but much more of a OS thread stuff....... im hoping wil have test done to end of the week.....