Improved masking options for human scanning

[tommyboy]

Hi all.  We've been using PhotoScan's built-in tool to get masks from background images (i.e. a second set of clean plate/empty background images taken right after the subject is photographed).  It does quite a good job, but seems to have the following issues:

1) the edge of the mask is often too generous in letting in the background, which lets through the white background; this is particularly problematic when creating textures for hair and areas where 'webbing' is likely (fingers, armpit, crotch)
2) it often ends up masking out dark portions of the subject if they are in the same spot as a camera lens in the background image
3) due to diffuse reflection of the subject on our white floor, the contact point of their feet with often includes a significant portion of the floor around their feet

So far we have been manually cleaning up these problem features using the PhotoScan tools, which ends up taking a good 60-90 seconds per photograph on average.  Obviously I'd like to take this number down to zero, so have been looking at alternative methods for background subtraction, which I could then feed into PhotoScan.  This project looked like a good option, but ended up having its own problems, and being rather slow:

http://docs.opencv.org/trunk/doc/tutorials/video/background_subtraction/background_subtraction.html

Is anyone else using external programs for mask generation?  I should know the answer to this but for some reason it is eluding me...

Thanks!

[admir]

Hi,
we are using our own tool for creating masks. It have two ways of extracting subject from background, from the inside of subject and from the outside. You can also specify treshold and luminosity factor. You setup all on 1st photo and background image and tool automatically apply masking/extraction for rest of images and creates .png masks. Ill post couple images tomorrow, its not the fastest but works well.

Admir

[chadfx]

Using some compositing-oriented tools can really help speed up the process; and give you better results. Something like Nuke, After Effects, etc. Even some automated pipelines in Photoshop could help, although not quite as ideal for dealing with variances between images.

You can extract mattes from clean bg's and also from any type of color; and combine various partial matte extractions to build up one perfect matte. Plus they have options to 'grow' or 'shrink' your matte if you need to trim or expand the matte. Those programs also have some nice color suppression tools, for when are finding something shot on blue or green screen is casting too much of that screen color back onto your subject.

They can handle roto tasks faster than Photoscan usually, too. (although Photoscan's magic lasso tool is pretty handy)

I'd love to see more of those tools inside Photoscan, but they are some not so small feature requests.

Cheers, -C

[markyboy]

Hi

Assume you've read my post Masking Hair from last week? I'm experimenting with not actually using background masking and just using the quick selection tool in Photoshop. Although with 100+ images it becomes a bit of a nightmare. This process also requires you to think about what what your actually masking, part of someone's hair can look very different from one angle to another and it's this thinking that's really slowing the process down for me? The only good thing it allows me to mask inside areas where previously photoscan would just leave a white patch.

Looking forward to a magic fix!

[Infinite]

1) the edge of the mask is often too generous in letting in the background, which lets through the white background; this is particularly problematic when creating textures for hair and areas where 'webbing' is likely (fingers, armpit, crotch)

It doesn't really matter how accurate the masks are in this regard as Photoscan still doesn't take into account masks during hole fill stage. Webbing will always occur until hole fill is taking masking into account. I believe this is a complex problem to solve.

It's just as fast to do your editing after dense point cloud reconstruction. Edit / delete the points directly in 3D, then build the mesh. Photoscan will still web but it will produce better results and will be faster than manually editing each mask image by hand.

[tommyboy]

1) the edge of the mask is often too generous in letting in the background, which lets through the white background; this is particularly problematic when creating textures for hair and areas where 'webbing' is likely (fingers, armpit, crotch)

It doesn't really matter how accurate the masks are in this regard as Photoscan still doesn't take into account masks during hole fill stage. Webbing will always occur until hole fill is taking masking into account. I believe this is a complex problem to solve.

It's just as fast to do your editing after dense point cloud reconstruction. Edit / delete the points directly in 3D, then build the mesh. Photoscan will still web but it will produce better results and will be faster than manually editing each mask image by hand.

Interesting point, thanks Lee!  It does indeed seem difficult to gauge how much time it is worth spending on masks, versus simply cleaning the model later.  When I clean the masks I certainly get a reduction in the amount/size of holes, and hole-filling by PS.  When you edit the point cloud directly for the same effect, you are effectively creating holes that will need to be filled.

So that's an important question, what are people's opinions: is the time spent tweaking PS's default masks better spent cleaning the model?

In either case, I'm mainly hoping there's a way to improve automated mask generation to the point that the answer to the above is definitely "clean the model".