Forum

Author Topic: v1.3 Performance Regression  (Read 6331 times)

feiko.lai

  • Newbie
  • *
  • Posts: 31
    • View Profile
v1.3 Performance Regression
« on: February 24, 2017, 07:01:47 AM »
Win 10 x64
Agisoft v1.3
Dual Xeon
R390x


We updated Agisoft from 1.2.6 to 1.3.0 and we observed performance regression on all our workstaions, e.g. for "build dense cloud",

1.2.6:
GPU: 269mil samples/sec CPU: 106 mil samples/sec
1.3.0:
GPU: 45mil samples/sec CPU: 15 mil samples/sec
 

Now we want to rollback to 1.2.6 but our projects are already saved in 1.3.0. Can we safely re-install 1.2.6 and open our projects?


feiko.lai

  • Newbie
  • *
  • Posts: 31
    • View Profile
Re: v1.3 Performance Regression
« Reply #1 on: February 24, 2017, 07:06:59 AM »
And "merge chunks"(6k images in total, 3 chunks, merge dense cloud and mesh) in 1.3 is incredibly slow

http://imgur.com/a/GdTal

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14813
    • View Profile
Re: v1.3 Performance Regression
« Reply #2 on: February 24, 2017, 10:54:55 AM »
Hello feiko.lai,

Performance values "mil samples/sec" cannot be compared with the previous version, as the appoach of the depth maps generation has been changed. You need to compare processing time for the same dataset processed in 1.2 with the same parameters.
Best regards,
Alexey Pasumansky,
Agisoft LLC

feiko.lai

  • Newbie
  • *
  • Posts: 31
    • View Profile
Re: v1.3 Performance Regression
« Reply #3 on: February 24, 2017, 11:30:46 AM »
Thank you Alexey,

As to my other questions (incredibly long chunk merging time and rolling back to 1.2), could you give me some suggestions?

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14813
    • View Profile
Re: v1.3 Performance Regression
« Reply #4 on: February 24, 2017, 12:20:14 PM »
Hello feiko.lai,

Can you provide the following information for each chunk being merged:
- number of cameras and image resolution,
- number of points in sparse cloud (used / total),
- number of depth maps (if any) and quality used,
- number of points in the dense cloud,
- number of polygons in mesh (if present).


Best regards,
Alexey Pasumansky,
Agisoft LLC

feiko.lai

  • Newbie
  • *
  • Posts: 31
    • View Profile
Re: v1.3 Performance Regression
« Reply #5 on: February 24, 2017, 09:07:00 PM »
24MP, multi-spectral image pairs

Other facts: http://imgur.com/a/jItfz ( those 3 expanded chunks)

While at the meantime, the estimated merging time keep growing http://imgur.com/a/n1BMi

 :'( :'(

Does it matter that we enabled "save depth map" before we start chunk merging?

We had quite a quick result with v1.2.6 and around 1000 images dataset before.

Thanks

feiko.lai

  • Newbie
  • *
  • Posts: 31
    • View Profile
Re: v1.3 Performance Regression
« Reply #6 on: February 27, 2017, 04:21:04 AM »
Please help!!

Now it will run for unbelievable time

http://imgur.com/a/4L9PJ

And all 128GB ram is eaten up !!!

http://imgur.com/a/5neeQ

zView

  • Newbie
  • *
  • Posts: 44
    • View Profile
Re: v1.3 Performance Regression
« Reply #7 on: February 27, 2017, 12:05:02 PM »
I see the same. After 91%, it takes ages to complete the dense point cloud when running in network mode.

feiko.lai

  • Newbie
  • *
  • Posts: 31
    • View Profile
Re: v1.3 Performance Regression
« Reply #8 on: February 28, 2017, 01:46:58 PM »
Please help, we can't proceed anymore.  :'( :'(

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14813
    • View Profile
Re: v1.3 Performance Regression
« Reply #9 on: March 01, 2017, 01:06:31 PM »
Hello feiko.lai,

We are trying to reproduce the issue with similar size large chunks.

Can you specify, if the project is saved in PSX or PSZ format?
Best regards,
Alexey Pasumansky,
Agisoft LLC

feiko.lai

  • Newbie
  • *
  • Posts: 31
    • View Profile
Re: v1.3 Performance Regression
« Reply #10 on: March 01, 2017, 01:32:58 PM »
Alexey,

The project was saved in psx format with depth map.




feiko.lai

  • Newbie
  • *
  • Posts: 31
    • View Profile
Re: v1.3 Performance Regression
« Reply #11 on: March 03, 2017, 01:10:35 PM »
Any update with the merging problem ? My team is still pending and making no progress.


Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14813
    • View Profile
Re: v1.3 Performance Regression
« Reply #12 on: March 05, 2017, 04:21:52 PM »
Hello feiko.lai,

We are running our tests to see the memory consumption of the merging operations for huge chunks.
Best regards,
Alexey Pasumansky,
Agisoft LLC

feiko.lai

  • Newbie
  • *
  • Posts: 31
    • View Profile
Re: v1.3 Performance Regression
« Reply #13 on: March 06, 2017, 05:44:49 AM »
Alexey,

Thank you, but the main problem is the merging is very slow, compared with a project with 2,000 images which was just a matter of 10 minutes, this took us around a week and eventually ate up all memory and could't finish.

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14813
    • View Profile
Re: v1.3 Performance Regression
« Reply #14 on: March 06, 2017, 12:10:18 PM »
Hello feiko,lai,

For quick test we've used three chunks with 500 millions points in the dense cloud each, and it took about 6-7 hours (including mesh merging) with the memory consumption peak about 50 GB. We have optimized the merging procedure, so it should be considerably faster now, but the memory consumption would be almost the same.

Then we've used the version with the fix to merge the original three chunks with the merging result, so 3 x 500 millions chunks and 1.5 billion cloud - it took less than one hour, and memory consumption was about 110 GB.
« Last Edit: March 06, 2017, 01:52:13 PM by Alexey Pasumansky »
Best regards,
Alexey Pasumansky,
Agisoft LLC