Forum

Author Topic: Automatic Masking.  (Read 39995 times)

Infinite

  • Sr. Member
  • ****
  • Posts: 366
    • View Profile
Automatic Masking.
« on: April 25, 2012, 04:57:26 PM »
I have already spoken to Alexey about this idea. But wanted to get it out there for others to ponder.

Automasking. A technique of using a set of images empty and a set with your subject in, full. If you think about it you can Mask between the 2 automatically as a difference. With or without.

I hope we can see something like this introduced into Agisoft. Alexey seems interested in it but wonder what others would think of how it could be introduced into the pipeline? perhaps as a tick box and input line field to point to the sessions empty set of images folder to process with at Align or Build stage? This would be ideal for video sequences and head or body captures, when using allot of images.



The idea of then being able to apply this in batch on dozens of images during Align or Geometry build, without the need to mask by hand or use a Chroma Key background with colour spill is an appealing one! at the moment my captures suffer allot with background objects (blobs) getting introduced in the final build model if not masked.

You can of course do Difference Matte in Aftereffects or Difference filter in Photoshop by hand but the results are a little tricky to get right but the idea is sound and it is rather time consuming!

If implemented it would then just be a matter of common practice to always take an empty set of images at the start of a capture session  :)

Interested in a discussion.

« Last Edit: April 25, 2012, 05:22:22 PM by Infinite »
_______________________________________________
I N F I N I T E
www.ir-ltd.net

FoodMan

  • Sr. Member
  • ****
  • Posts: 477
    • View Profile
Re: Automatic Masking.
« Reply #1 on: April 25, 2012, 05:50:10 PM »
Well that would be an awesome feature..

+1

f/

Kjellis85

  • Full Member
  • ***
  • Posts: 220
  • Archaeological field supervisor
    • View Profile
    • Contact information at University of Tromsø
Re: Automatic Masking.
« Reply #2 on: April 25, 2012, 07:51:24 PM »
I think this is a great idea, not only for people, but for artifacts as well. If you could automask like this, then you would not need to move around the object, saving a lot of time.

+1 from me :)

tezen

  • Jr. Member
  • **
  • Posts: 68
    • View Profile
Re: Automatic Masking.
« Reply #3 on: April 25, 2012, 08:19:17 PM »
Hello Infinite and Alexey!

+1.
Your option is quite a good idea for still-standing cameras. If the camera-position is moving, then Alexey has to program a routine which will align images (by background).

I throw another idea of automasking into this thread for discussion:
Automasking by point-cloud and region-box ...here shown in "hand-made mode"!
- Calculate a point-cloud (10000 points and more) without a mask -  the cameras are usually better aligned in my eye.
- If you view an specific foto and right click on it for "reset view" you will see through the calculated camera.
- Make a screenshot for the imagesoftware of your choice.
- Colorize the points (ONLY those which are inside the region-box) into white and everything else into black.
- Duplicate the image and blur it (bigger blur for ~10000 points and smaller blur for details or 30000+ points).
- Tweak the luminance at your choice and turn that greyscale-image picture into a black&white one.
- Make that generated mask a little bit smaller (big as the blur in correspondence to the luminance-settings)
- Voila?!

If that could be done automatically (one click on a button called "Automask by PC/RB"!) it would be less work in most cases!

Greetings
tezen
« Last Edit: April 25, 2012, 08:20:55 PM by tezen »

mala

  • Full Member
  • ***
  • Posts: 109
    • View Profile
Re: Automatic Masking.
« Reply #4 on: April 25, 2012, 11:23:29 PM »
Hi Lee,

 Can see where you are going with this...and indeed it would be very handy ;)

would love to see it implemented

Cheers,
Merry

JMR

  • Hero Member
  • *****
  • Posts: 522
    • View Profile
Re: Automatic Masking.
« Reply #5 on: May 03, 2012, 08:54:40 PM »
Your idea sounds smart and I'd like to have a tool for this too, but having said that, I find this request too much focused on an specific need which may not be of general use. I guess it could be done outside (Photoshop macro could fit) if you managed to generate mask files suitable to be imported into Photoscan project and applied to photos by means of scripting.

regards,

Jos?


Infinite

  • Sr. Member
  • ****
  • Posts: 366
    • View Profile
Re: Automatic Masking.
« Reply #6 on: May 03, 2012, 09:00:12 PM »
Your idea sounds smart and I'd like to have a tool for this too, but having said that, I find this request too much focused on an specific need which may not be of general use. I guess it could be done outside (Photoshop macro could fit) if you managed to generate mask files suitable to be imported into Photoscan project and applied to photos by means of scripting.

regards,

Jos?

How is it a specific need? It's a general workflow that could work as a pipeline rule, always. Masking images both vastly improves Geometry Build and Texture quality across the board (saying this from experience). Masking and importing by hand is both time consuming and intensive, especialy if you have to work with bloated .png's or .tiff's.

This workflow would be ideal for face capture, full body capture, props and model/miniature capture. For both "3D" and "4D" input.
_______________________________________________
I N F I N I T E
www.ir-ltd.net

JMR

  • Hero Member
  • *****
  • Posts: 522
    • View Profile
Re: Automatic Masking.
« Reply #7 on: May 04, 2012, 01:52:07 AM »
I know your work, and I sincerely admire it.
but I still think that your case is not that common. I might have misunderstood something but
these are its specifics to my understanding.
1. You have more than one camera. (how could the more common single camera user do the trick of taking all photos with and without subject?).
2. Your subject is not only separated from the background, but also removable.
3. You would probably like to mask thousands of photos that share the same attitude and background to subtract. It is indeed a reasonable desire because it sounds boring... but still rather specific imho.
I wish I could say I have the same needs as you because that would mean I have your flock of cameras and your magic hopefully.

Let me point that in your setup there are interesting invariants that could be exploited if photoscan allowed that, and here (again imho) it could make a little more sense to ask for such a feature. As long as your setup is fixed in all aspects but the subject. Why the need of calculating external parameters for each set of simultaneous photos. and why not fix the bounding box too. You should be allowed to do it just once in that cases and go directly to dsm phase.

For automating  masking, let me think aloud about Tezen's approach.
suppose you have the sparse cloud, then you can resize and orient the bounding box to fit your subject... back-project the 3D points inside the box to all images and think about this new planar cloud onto your images. how could we convert that cloud into a mask?
Tezen suggest a trick that works as if those pixels were binarized and expanded until they fill a certain  region but that would probably exceed subject's boundaries (which is bad) and leave holes where little feature points were found... we need a better hint.

Convex-hull is my first bid

Here starts the game: share your thoughts

my kindest regards

Infinite

  • Sr. Member
  • ****
  • Posts: 366
    • View Profile
Re: Automatic Masking.
« Reply #8 on: May 04, 2012, 02:07:53 AM »
I know your work, and I sincerely admire it.
but I still think that your case is not that common. I might have misunderstood something but
these are its specifics to my understanding.
1. You have more than one camera. (how could the more common single camera user do the trick of taking all photos with and without subject?).
2. Your subject is not only separated from the background, but also removable.
3. You would probably like to mask thousands of photos that share the same attitude and background to subtract. It is indeed a reasonable desire because it sounds boring... but still rather specific imho.
I wish I could say I have the same needs as you because that would mean I have your flock of cameras and your magic hopefully.

Let me point that in your setup there are interesting invariants that could be exploited if photoscan allowed that, and here (again imho) it could make a little more sense to ask for such a feature. As long as your setup is fixed in all aspects but the subject. Why the need of calculating external parameters for each set of simultaneous photos. and why not fix the bounding box too. You should be allowed to do it just once in that cases and go directly to dsm phase.

For automating  masking, let me think aloud about Tezen's approach.
suppose you have the sparse cloud, then you can resize and orient the bounding box to fit your subject... back-project the 3D points inside the box to all images and think about this new planar cloud onto your images. how could we convert that cloud into a mask?
Tezen suggest a trick that works as if those pixels were binarized and expanded until they fill a certain  region but that would probably exceed subject's boundaries (which is bad) and leave holes where little feature points were found... we need a better hint.

Convex-hull is my first bid

Here starts the game: share your thoughts

my kindest regards

1. If the user, who has only 1 Camera uses a white, black or colour background the same thing could be useful. A way to automate the masking procedure. Although this could be done with a python script as well in Agisoft.

All this could be automated, to a point, outside Agisoft but it is not trivial for allot of images and damn hard work, not just boring but physically and mentally demanding. For example I have recently just finished processing 270 expressions of 3 actors for a client. The task is taxing to say the least.

I've been contacted by 5 companies in the last few weeks who are talking about their own multi-camera system with agisoft and how they can improve their workflow. I think you will find multi camera setups will start to become the norm for most people and companies who are serious about capture. Even if it's just a stereo pair, it's still useful. 1 Shot with subject, 1 shot without. Even if they Chroma key, this idea would still work in that scenario. Rotate and repeat. Automask.

I can understand for users with 1 camera they may not need it but for license payers who have shelled out for the PRO version, like myself, it would be ideal. Thanks to Alexey's super hard work Agisoft is already improving greatly even with the last few upgrades thanks to his innovation. This feature along with 4D processing would place Agisoft into another class of production proven software.

The system here is not always fixed. Automasking would shave off hours, possibly days of manually labour. Per Pose!
_______________________________________________
I N F I N I T E
www.ir-ltd.net

tezen

  • Jr. Member
  • **
  • Posts: 68
    • View Profile
Re: Automatic Masking.
« Reply #9 on: May 04, 2012, 10:03:51 AM »
Hello!

Now here are the four ideas of this thread:
1. Masking by greenscreen (black, white or colored background)
2. Masking by background (only for static cams)
3a. Masking by pointcloud & bounding-box
3b. ...with a convex-hull

Let me throw in one more idea:
Masking by Focus. Like allmost every cam got a algorythm for Auto-Focus the same algorythm could be used to seperate the areas with fine details and that blurry ones. Maybe only for generating a convex-hull around it.

Different automasking-options for different situations would be good.

Greetings
tezen

FoodMan

  • Sr. Member
  • ****
  • Posts: 477
    • View Profile
Re: Automatic Masking.
« Reply #10 on: June 24, 2012, 12:31:04 PM »
Hello Alexey.... are you working on a solution...?  8)

thanks
f/

Infinite

  • Sr. Member
  • ****
  • Posts: 366
    • View Profile
Re: Automatic Masking.
« Reply #11 on: June 24, 2012, 01:38:07 PM »
Hello Alexey.... are you working on a solution...?  8)

thanks
f/

It's already been implemented, I've been lucky enough to test it, it works amazingly well! soon to be released in the next builds I think. Not sure it it's Pro or Standard. The AgiSoft guys can say.
_______________________________________________
I N F I N I T E
www.ir-ltd.net

FoodMan

  • Sr. Member
  • ****
  • Posts: 477
    • View Profile
Re: Automatic Masking.
« Reply #12 on: June 24, 2012, 01:53:34 PM »
Hello Lee... thank you very much for the excellent news !... Can't wait to test it..  8)

FoodMan

  • Sr. Member
  • ****
  • Posts: 477
    • View Profile
Re: Automatic Masking.
« Reply #13 on: June 25, 2012, 08:20:57 AM »
btw, maybe just a little question if you don't mind... what is the solution used...? the first one you purposed .?

Thanks
f/

Infinite

  • Sr. Member
  • ****
  • Posts: 366
    • View Profile
Re: Automatic Masking.
« Reply #14 on: June 25, 2012, 10:44:24 AM »
btw, maybe just a little question if you don't mind... what is the solution used...? the first one you purposed .?

Thanks
f/

Yes the first one, Difference Masking.
_______________________________________________
I N F I N I T E
www.ir-ltd.net