Forum

Author Topic: Cut and trim by masks combined with Intelligent Hole Filling  (Read 26117 times)

Infinite

  • Sr. Member
  • ****
  • Posts: 366
    • View Profile
Cut and trim by masks combined with Intelligent Hole Filling
« on: November 22, 2012, 04:14:01 PM »
Just an idea to improve final silhouette of scans by cutting and trimming feature after geometry build. Perhaps with tolerance settings from 0-100?

If we could generate super accurate masks, either manually or automatic and have the ability to allow Agisoft to then trim and cut the scans using the combined masks? Like an extra step "Trim and Cut" sort of like a hack similar to silhouette reconstruction. This auto trimming process could save hours of manual sculpting, cutting and clean up and also improve scan quality in some areas by chopping out noise and blobs. For example I can get really bad areas under arm pits, between legs and fingers on full body data. Yet the background masks can be super sharp and clean and be perfect for cutting.
« Last Edit: July 24, 2013, 12:11:28 PM by Infinite »
_______________________________________________
I N F I N I T E
www.ir-ltd.net

Wishgranter

  • Hero Member
  • *****
  • Posts: 1202
    • View Profile
    • Museum of Historic Buildings
Re: Cut and trim by masks
« Reply #1 on: November 22, 2012, 04:45:43 PM »
+1 from me toooo.....
----------------
www.mhb.sk

tezen

  • Jr. Member
  • **
  • Posts: 68
    • View Profile
Re: Cut and trim by masks
« Reply #2 on: November 22, 2012, 05:16:17 PM »
+1
...that could improve abituary model generation a lot!

Mr_Curious

  • Jr. Member
  • **
  • Posts: 86
    • View Profile
Re: Cut and trim by masks
« Reply #3 on: November 22, 2012, 06:16:10 PM »
+1

We really need this functionality badly.  Currently, if you produce a lot of models the clean-up required is workflow killer.  Any improvement in this particular area would be most appreciated :-)

Greetings,

Mr. Curious

augeaz

  • Newbie
  • *
  • Posts: 29
    • View Profile
Re: Cut and trim by masks
« Reply #4 on: November 24, 2012, 06:58:45 PM »
+1

I am new to this but already full hands of zbrushing excess polygons.

Infinite

  • Sr. Member
  • ****
  • Posts: 366
    • View Profile
Re: Cut and trim by masks
« Reply #5 on: November 24, 2012, 09:52:48 PM »
Great example here:

Notice how crisp and sharp the mask data is? event the point data is clean. Yet the final build contains allot of blobbing and webbing in join areas  :'(





We should be able to use this mask and point data to our advantage  :)

« Last Edit: November 24, 2012, 09:55:14 PM by Infinite »
_______________________________________________
I N F I N I T E
www.ir-ltd.net

tezen

  • Jr. Member
  • **
  • Posts: 68
    • View Profile
Re: Cut and trim by masks
« Reply #6 on: November 25, 2012, 02:25:14 PM »
Hello!

All in all its "just" a boolean-operation, which you could do manually in other software:

1. Generate a polyplane from the non-masked-area and thicken it to become a polygon. F.E. in ZBrush you can  "make 3d" out of an black&white-alpha with settings at MRes 512 or higher and minimal smooth + MDep .
2. That polygon have to face/rotate/positioned onto the camera where the mask is coming from. You could do it in any modelling program which is able to load the camera-positions calculatet by PS.
3. Resize that polygon in its own Z-direction until it hits nearly the middlepoint of the camera and covers the whole object.
4. Resize the polygons Z-direction-ends in their own X&Y-axis until they match the 2D-mask of the camera.
5. Do a boolean-operation to cut the polygon out of your PS-mesh.

In theory it seems to be relative simple. But its not easy to write some sort of software for that process (especially boolean-operations on multimillion-poly-meshes).

Greetings
tezen

Mr_Curious

  • Jr. Member
  • **
  • Posts: 86
    • View Profile
Re: Cut and trim by masks
« Reply #7 on: November 25, 2012, 03:42:47 PM »
Hello!

All in all its "just" a boolean-operation, which you could do manually in other software:

1. Generate a polyplane from the non-masked-area and thicken it to become a polygon. F.E. in ZBrush you can  "make 3d" out of an black&white-alpha with settings at MRes 512 or higher and minimal smooth + MDep .
2. That polygon have to face/rotate/positioned onto the camera where the mask is coming from. You could do it in any modelling program which is able to load the camera-positions calculatet by PS.
3. Resize that polygon in its own Z-direction until it hits nearly the middlepoint of the camera and covers the whole object.
4. Resize the polygons Z-direction-ends in their own X&Y-axis until they match the 2D-mask of the camera.
5. Do a boolean-operation to cut the polygon out of your PS-mesh.

In theory it seems to be relative simple. But its not easy to write some sort of software for that process (especially boolean-operations on multimillion-poly-meshes).

Greetings
tezen

True enough.  However, all this manual labour is very time consuming and more or less defeats the purpose of what is trying to be achieved here which is clean meshing results, quickly without the need for manual human intervention which is a workflow killer.

Greetings,

Mr. Curious

Infinite

  • Sr. Member
  • ****
  • Posts: 366
    • View Profile
Re: Cut and trim by masks
« Reply #8 on: November 25, 2012, 06:54:41 PM »
Hello!

All in all its "just" a boolean-operation, which you could do manually in other software:

1. Generate a polyplane from the non-masked-area and thicken it to become a polygon. F.E. in ZBrush you can  "make 3d" out of an black&white-alpha with settings at MRes 512 or higher and minimal smooth + MDep .
2. That polygon have to face/rotate/positioned onto the camera where the mask is coming from. You could do it in any modelling program which is able to load the camera-positions calculatet by PS.
3. Resize that polygon in its own Z-direction until it hits nearly the middlepoint of the camera and covers the whole object.
4. Resize the polygons Z-direction-ends in their own X&Y-axis until they match the 2D-mask of the camera.
5. Do a boolean-operation to cut the polygon out of your PS-mesh.

In theory it seems to be relative simple. But its not easy to write some sort of software for that process (especially boolean-operations on multimillion-poly-meshes).

Greetings
tezen

I absolutely agree with Mr. Curious.

Thanks for taking the time to think of a workflow but trust me I have tried it all. I've been using ZBrush since 2.5 back in 2003-2004. I know most of the tricks. The point is the labor and time involved, if you multiply that by 100's of full body poses, per person! it gets very laborious.

Also the biggest issue isn't the cutting/trimming, this is easy, it's the filling in. We need this to be accurate and volumetric during or after geometry build in arbitrary mode. Human intervention is no good.

Again thanks for thinking of ideas but not suitable in this case.
_______________________________________________
I N F I N I T E
www.ir-ltd.net

JMR

  • Hero Member
  • *****
  • Posts: 504
    • View Profile
Re: Cut and trim by masks
« Reply #9 on: November 25, 2012, 09:51:49 PM »
I tend to agree this could be a very good idea but with exceptions (so mine is +0.99). I've noticed that oversizing masks helps to get better geometry. I mean that if you mask object zones nearby the very silhouette or intentionally mask image regions corresponding to object's faces that are very oblique to point of view, this improves geometry and help to get cleaner model. In this case if you trim the model by-mask, you'll actually destroy some good geometry parts ... :(
So, a good idea that needs to be fine tuned and carefully engineered

Infinite

  • Sr. Member
  • ****
  • Posts: 366
    • View Profile
Re: Cut and trim by masks
« Reply #10 on: November 25, 2012, 10:06:32 PM »
I tend to agree this could be a very good idea but with exceptions (so mine is +0.99). I've noticed that oversizing masks helps to get better geometry. I mean that if you mask object zones nearby the very silhouette or intentionally mask image regions corresponding to object's faces that are very oblique to point of view, this improves geometry and help to get cleaner model. In this case if you trim the model by-mask, you'll actually destroy some good geometry parts ... :(
So, a good idea that needs to be fine tuned and carefully engineered

That's why as a tick option it could be very powerful. Either for during Geometry build or after as a process to cut/trim and fill correctly based on masks. Not a forced process but up to the user to utilize.
_______________________________________________
I N F I N I T E
www.ir-ltd.net

FoodMan

  • Sr. Member
  • ****
  • Posts: 477
    • View Profile
Re: Cut and trim by masks
« Reply #11 on: November 26, 2012, 11:03:58 AM »
+1

James

  • Hero Member
  • *****
  • Posts: 748
    • View Profile
Re: Cut and trim by masks
« Reply #12 on: November 26, 2012, 12:34:38 PM »
Slight variation on the same theme, but that does not rely on having great masks (mine are rarely precise enough for this!)

After building geometry, using the freeform (or other) selection tool to manually select an ugly blob, and then shift-del to delete and automatically fill the hole, or ctrl-delete to delete and open a dialog box (or not) for filing the hole taking into account surrounding geometry curvature.

Either using this technique or those above will be limited when you can see correct geometry behind the area you are trying to cut/trim. To resolve this you would need to implement a manipulable 'editing limit box' that restricted any trimming to geometry within it.

But maybe this is getting too far into mesh cleaning and far away from 3d photo reconstruction?

Infinite

  • Sr. Member
  • ****
  • Posts: 366
    • View Profile
Re: Cut and trim by masks
« Reply #13 on: November 26, 2012, 10:08:43 PM »
Slight variation on the same theme, but that does not rely on having great masks (mine are rarely precise enough for this!)

After building geometry, using the freeform (or other) selection tool to manually select an ugly blob, and then shift-del to delete and automatically fill the hole, or ctrl-delete to delete and open a dialog box (or not) for filing the hole taking into account surrounding geometry curvature.

Either using this technique or those above will be limited when you can see correct geometry behind the area you are trying to cut/trim. To resolve this you would need to implement a manipulable 'editing limit box' that restricted any trimming to geometry within it.

But maybe this is getting too far into mesh cleaning and far away from 3d photo reconstruction?

The hole filling is the issue. It depends how this is done, for example in ZBrush you can remove excess and fill but the fill (or close) will produce worse results! like a webbing effect or larger blob. We need an intelligent hole filling perhaps taking into account point cloud or mask data as a guide.
_______________________________________________
I N F I N I T E
www.ir-ltd.net

tezen

  • Jr. Member
  • **
  • Posts: 68
    • View Profile
Re: Cut and trim by masks
« Reply #14 on: November 27, 2012, 08:45:56 PM »
Hey!

...as described above it would be nice to have a new reconstruction-method called "mask" (besides abituary, heightfield and pointcloid). Like ZBrushes shadowbox in more than 3 directions and within perspective distortion. The result should look like basic carved wood and even much better than low-poly or pointcloud-reconstruction. You could use that wood-like-object for boolean-operations inside other software (@Infinite&Mr.Curious: ...and you only have to do it once per object - by macroscript inside ZBrush ;) ). But thats not automatic enough!

Measuring distances between an high-abituary-reconstruction and that object could be used for calculating critical zones (=inner edges - I hope you know what I mean!). As the distance increases the abituary-reconstruced points should move in their own Z-direction down (or up) to the wood-like-object. For correct shapes (non-overlapping) the normals of those points have to be smoothened by their neighbors.

The problem with the critical zones (=inner edges) depends mostly on the smooth-algorythm (beneath lightning) I think. Using a sharp-algorythm most polys stay in the right position. Maybe a mix between the sharp and the smooth algorythm will be the solution (sharp at zones with lots of 3D-information and smooth at zones without many info and/or lots of holes)? It would increase the details alot. Look for yourself and take a test with photos of your choice. You could use the same generated depthmaps (BTW: In Batch-mode theres a missing option for turning off rebuild depthmaps).

A mix between mild and aggressive depth filtering (aggressive at zones with lots of 3D-info and mild at zones without) would increase details at pre-processing-stage? Dont know! But individual settings depending on the objects area (data-ammount, edgy or soft surface etc.) instead of static settings (p1,p2 which are now calling mild,moderate&aggressive or sharp (=0) and smooth (=1 - gaussian algorythm?)) could increase the quality of the result. These are just ideas - maybe someone is inspired by these!

Greetings
tezen