Forum

Author Topic: RAM vs SWAP - can the job be done of you have enough swap  (Read 8375 times)

andrekje

  • Jr. Member
  • **
  • Posts: 63
    • View Profile
RAM vs SWAP - can the job be done of you have enough swap
« on: October 09, 2015, 04:45:24 PM »
If a given job requires 64GB of RAM , can it be processes by 64GB RAM  , or 32GB RAM+ 32GB SWAP partition  , or even 16GB RAM +48GB swap ?

Sure, the SSD swap will work much slower than RAM, but how far can this be stretched ?  - what if I would stitch lots of pictures that ,in theory, would require ~1TB or RAM ?

ferdywallinx

  • Newbie
  • *
  • Posts: 28
    • View Profile
Re: RAM vs SWAP - can the job be done of you have enough swap
« Reply #1 on: December 23, 2015, 01:09:51 PM »
Hi,

what was in the end your solution? Did you try multiple things? I'm struggling with the same issue.

Lambo

  • Full Member
  • ***
  • Posts: 167
    • View Profile
Re: RAM vs SWAP - can the job be done of you have enough swap
« Reply #2 on: December 23, 2015, 01:42:23 PM »
I can only tell you that swapping can slow down your work 10 times or more. If you can you must avoid the swapping.
Besides the fact that the computer becomes unusable during that time if you need to do anything else.

Leo

Pixel UAV

  • Newbie
  • *
  • Posts: 15
    • View Profile
Re: RAM vs SWAP - can the job be done of you have enough swap
« Reply #3 on: January 12, 2016, 03:06:05 AM »
we have only 64gb of RAM on our processing computer, but use a 256GB SSD dedicated to SWAP.  Works really well and is cheap.  We process jobs of 2000 24mp photos with no problems.

stihl

  • Sr. Member
  • ****
  • Posts: 410
    • View Profile
Re: RAM vs SWAP - can the job be done of you have enough swap
« Reply #4 on: January 12, 2016, 03:47:05 PM »
2k 24mp should fit within 64gb's depending on the used resolution of the source images..

holgar

  • Jr. Member
  • **
  • Posts: 62
    • View Profile
Re: RAM vs SWAP - can the job be done of you have enough swap
« Reply #5 on: January 12, 2016, 04:32:33 PM »
I am at the moment calculating 188 18MP Images with Ultrahighsetting. My Computer with 64GB went into swapping. I really wonder why. My Memory monitor tells me I have 18GB free memory. I have the feeling something is wrong with the latest Beta.

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14813
    • View Profile
Re: RAM vs SWAP - can the job be done of you have enough swap
« Reply #6 on: January 12, 2016, 05:00:46 PM »
I am at the moment calculating 188 18MP Images with Ultrahighsetting. My Computer with 64GB went into swapping. I really wonder why. My Memory monitor tells me I have 18GB free memory. I have the feeling something is wrong with the latest Beta.
Hello holgar,

What type of the object you are reconstructing and at what step the processing is stuck? Is it already depth maps filtering?

Maybe you can provide the screenshot of performance tab of the task manager (if you are on Windows) and current console output (if it's still open now).
Best regards,
Alexey Pasumansky,
Agisoft LLC

holgar

  • Jr. Member
  • **
  • Posts: 62
    • View Profile
Re: RAM vs SWAP - can the job be done of you have enough swap
« Reply #7 on: January 12, 2016, 05:21:10 PM »
I am at the moment calculating 188 18MP Images with Ultrahighsetting. My Computer with 64GB went into swapping. I really wonder why. My Memory monitor tells me I have 18GB free memory. I have the feeling something is wrong with the latest Beta.
Hello holgar,

What type of the object you are reconstructing and at what step the processing is stuck? Is it already depth maps filtering?

Maybe you can provide the screenshot of performance tab of the task manager (if you are on Windows) and current console output (if it's still open now).

Hi Alexey, I attached Screenshots from my aktivity monitor software ( i am on osx). The Swapping happens pretty early in the Denseclould generation. But I am not sure now at which point. The Progressbar is not really reflecting the reality. It stayed around Six Hours remaining for several Hours. Now I am at 18.5 Hours.

I am scanning a sneaker. I have very very sharp images. I set the Keypoint limit to 500.000 and the tiepoint limit to 0.
The result of alignment is very accurate and I get 1.9 Million Tiepoints (unfiltered). The resulting Densecloud will have 210 Million Points ( I know that because I proccesed the images already. I just forgot to adjust the boundingbox so a little from the top of the shoe was cut of. Thats why I need to process again :-()




« Last Edit: January 12, 2016, 05:26:49 PM by holgar »

holgar

  • Jr. Member
  • **
  • Posts: 62
    • View Profile
Re: RAM vs SWAP - can the job be done of you have enough swap
« Reply #8 on: January 13, 2016, 09:46:19 AM »
sorry, I think I found the issue. My Memory monitor is showing "incative Memory" as free. So it seems there is a lot of reserved memory for Photoscan which is not actually used but also not free. This causes swapping. I am not so into the technical stuff like this. But I have the feeling Photoscan should not be swapping if there is so much inactive memory?

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14813
    • View Profile
Re: RAM vs SWAP - can the job be done of you have enough swap
« Reply #9 on: January 13, 2016, 11:56:11 AM »
Hello holgar,

If you can provide us the project file and the dataset for internal tests, we can check the memory consumption during dense cloud generation step.

High memory consumption and long processing time may be related to the excessive overlap, when there are a lot of overlapping images of different scale. For example, when some zones are present on dozens of photos, usually it may happen if the object is fully present on multiple images.
Best regards,
Alexey Pasumansky,
Agisoft LLC

holgar

  • Jr. Member
  • **
  • Posts: 62
    • View Profile
Re: RAM vs SWAP - can the job be done of you have enough swap
« Reply #10 on: January 13, 2016, 12:26:46 PM »
Hello holgar,

If you can provide us the project file and the dataset for internal tests, we can check the memory consumption during dense cloud generation step.

High memory consumption and long processing time may be related to the excessive overlap, when there are a lot of overlapping images of different scale. For example, when some zones are present on dozens of photos, usually it may happen if the object is fully present on multiple images.

Hi Alexey, unfortunatly its a high confidential project and I can not share Data. I did not know that more overlap causes more memory consume. Will consider this in my next Projects. I thought the more overlap the better and choose to take better more than less images. My overlap in the Chunkinfo is at 3.2. Do you think that is to high? What would be a good vallue?

andrekje

  • Jr. Member
  • **
  • Posts: 63
    • View Profile
Re: RAM vs SWAP - can the job be done of you have enough swap
« Reply #11 on: January 13, 2016, 12:40:10 PM »
this thread moved a little away from the original subject (which is forgotten?)
Basically, I have never seen a job fail due to memory, after consuming all of SSD Swap.
Also, be it 64GB or 256GB of RAM, I think we are merely pushing the problem ahead for bigger jobs.
Should't it be possible to only process a chunk at the time in memory, then dump to swap/temp ?
If I can stitch 1km2  , then I should be fine to stitch 1000km2 , by processing an area, unloading most of it, keeping the border area to next chunk, then process that ?

Such processing would only require more tmp/swap , that can be fast on SSD stripe set with proper controllers, it would  remove the "never enough RAM" in a world where we only generate more data as sensors improve.

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14813
    • View Profile
Re: RAM vs SWAP - can the job be done of you have enough swap
« Reply #12 on: January 13, 2016, 12:45:02 PM »
Hello holgar,

I can suggest to exclude (disable after alignment, for example) the overview images that contain the whole object in the frame, if there are sufficient close-up images with good overlap. Otherwise, such general view images will have overlap with a lot of close-up images and the depth filtering will take a lot of time, while the presence of such images doesn't seem to have great effect on improving mesh or texture quality, due to lower effective resolution.
But since I'm not sure about your shooting scenario and camera positions, I don't know if there are may similar images, but you can check it and if there are quite a lot, it might be the most probable reason of the long processing time.
Best regards,
Alexey Pasumansky,
Agisoft LLC

holgar

  • Jr. Member
  • **
  • Posts: 62
    • View Profile
Re: RAM vs SWAP - can the job be done of you have enough swap
« Reply #13 on: January 13, 2016, 12:48:40 PM »
Hello holgar,

I can suggest to exclude (disable after alignment, for example) the overview images that contain the whole object in the frame, if there are sufficient close-up images with good overlap. Otherwise, such general view images will have overlap with a lot of close-up images and the depth filtering will take a lot of time, while the presence of such images doesn't seem to have great effect on improving mesh or texture quality, due to lower effective resolution.
But since I'm not sure about your shooting scenario and camera positions, I don't know if there are may similar images, but you can check it and if there are quite a lot, it might be the most probable reason of the long processing time.

I use a turntable with the same camera distance for all images. Maybe next time I try to make less overlapping and see how that works.
Sorry for getting off topic a bit.