Using physical markers in "background" to improve alignment

[DCK]

Several posts have suggested that when alignment is difficult using the object only, having physical markers in the "background" can improve alignment. For example, you could place the object on a newspaper, in which case the text of the newspaper, unmasked, would be incorporated into the alignment calculations.

I am trying to figure out how to make this work for my imaging sequence. I'm making models of skulls (and other bones). I place the skull on a turntable and rotate the turntable to surround the skull with images. I surround the skull with images on 3-6 axes. Some of the skulls are very undifferentiated in terms of color and topography, and thus difficult to get good alignment.

To create physical markers, I can tape a page of newspaper onto the surface of the turntable. Then I can align separately for each axis around which I have taken images. The remaining steps are unclear to me.

Ultimately, I want a model of the skull without the newspaper. But each aligned chunk will contain a model that includes the newspaper. So, when do I merge chunks? And when and how do I remove the newspaper reconstruction from my model?

Thanks again.

[Kiesel]

 For the professional version it should go like this (I don't own it):

1. align images for every rotation (around one axis, you have done this)
2. mask the newspaper out with masks (or edit the model later see step 4)
3. build model for every rotation in a separate chunk
4. edit the model in every chunk, so that all uninteresting parts are erased
5. set markers on identical points in every chunk
6. align chunks with the help of the markers
7. merge chunks

For the Standard version you could try to align the chunks without markers but with masking all newspaper out (step 6).

You can find a very detailed discription for Paleontology in: "PHOTOGRAMMETRY IN PALEONTOLOGY – A PRACTICAL GUIDE" http://www.jpaleontologicaltechniques.org/pasta3/JPT%20N12/Pdf/JPT_n012_Jul.pdf

Karsten

[bigben]

If you can raise the skull a little above the turn table it's easier to select and remove the unwanted points.

Skulls are relatively complex objects, and I'd probably take more shots by moving the camera around to ensure sufficient overlap inside the orbits and other complex shapes. Make sure you fill the frame as much as possible with the skull, or even move in so close that you're only photographing part of the skull.  Unless the skull is extremely smooth I'm a little surprised that it doesn't work well. What resolution is the camera? Can you post a sample photo?

[DCK]

I've gotten good results with many specimens, but there are a few that seem to escape me.

The first type of problem is with specimens that have a very undifferentiated vault. (See Image 1). My concern is that these difficult specimens look a lot like many one would encounter collecting data using a natural history museum's collections. The preparation process of a deceased animal to get down to its bones often results in very white, very smooth specimens.

The second type of problem is with small specimens (for example, a skull with a max. breadth < 3 cm, as in Image 2). With these, I can't fill the frame with the specimen and still hold the specimen in focus. By the time I back up far enough to get the whole specimen in focus at f13, the specimen itself fills only a small percentage of the frame. In fact, this is a problem with both the Image 1 and Image 2 specimens. I need to be 2 feet back to get a 2-inch focal depth (approx. antero-posterior length of Image 2 specimen), and almost three feet back to get 4 inch focal depth (approx. antero-posterior length of Image 1 specimen).

FYI I can model the specimen in Image 2 with good overall geometry, but the vault winds up noisy and the model only weakly expresses important landmarks like sutures.

I'm using a Nikon D5000 (12.1mpx). I'm shooting JPEGs with quality level "fine." Tomorrow I'm going to give it a try with a Canon 21 mpx camera (Mark-somethingorother).

Note: attached images compressed.

[bigben]

OK, so my comments were partly based on our usage of a 21mpx camera as well. I've often been surprised at how much extra this picks up.  I'm sure someone will eventually chime in with focus stacking to get the extra depth of field so I'll throw in a slight twist on that theme (based on limited testing).

Firstly, you don't need the entire object to be in focus particularly once the surface is angled beyond 60° from the camera. It will eventually come into focus as you go around the object. You can compensate for a shallower depth of field by shooting with less separation between photos.

In some cases, with complex geometry, you may need something at the far end of the object to be in focus because it will be the only orientation of the object where you can see the surface. In this instance you can take multiple shots at different distances  and either use focus stacking OR just add all of the images into PS without stacking. The latter has worked for me although the objects did not have as much depth as you have in a skull. 

For the skull in image 2, I'd probably sit it face up on a cushion and move the camera around  to get images from lots of different angles as most of the complexity is in the facial portion. I've found PS seem to work better with multiple linear passes (see attached diagram) rather than a circle. You get more overlap in the foreground (where reprojection error and reconstruction uncertainty tend to be much lower) After that, just rotate it around to different orientations to capture the other surfaces.

Instead of newspaper I'd suggest a piece of dark grey material as it will be easier to mask out and the texture will still provide ample detail. Don't try and keep it perfectly flat though or the pattern may be too contiguous.

The second image shows a small bronze turtle on a "conservation-friendly" cushion perched on its side to get the lateral edges.  In this particular case I got a detailed model of all of the cushion as well as the object (without masking)

[DCK]

It's good to hear someone else say that perfect focus across the whole object isn't essential! I've suspected as much, and asked about it in prior posts, but no one has responded.

True image stacking isn't an option for me. My job here is to come up with a routine that is easily repeatable and pretty efficient--a method that could be used to digitize an entire collection of vertebrate skeletal remains (or at least the elements deemed most important). So I'd prefer not to add image-stacking software, along with the added time to process stacked images in that software, to the routine.

However, the idea of taking shots at different distances is appealing. I have an 18-55mm lens. I assume I could accomplish what you're recommending simply by changing my focal range???

By the way, the real complexity in the skulls is on the base. PS does a great job of picking up this topography. It's the smooth areas, ie the vault, where the results are compromised.

Also, Kiesel, thanks for the link to that article. Good stuff.

[ThomasVD]

Rather than manually masking the newspaper or marker in each picture, try generating a model of each chunk (at least to the "mesh" stage), then just select the newspaper in the mesh and delete it from the model.
Afterwards to go Tools -> Import -> Import Masks, and choose "from model" in order to mask the newspaper in all the pictures from that chunk.

This is a particularly useful tool if you want to create models of the skulls without the turntable, but also containing the bottom of the skulls. Generate a model of the upper-side of the skull, delete the turntable in the model and import masks. Then do the same for the bottom. Finally re-process all images together with the masks in place.

Also about the blurry background: I've had good results simply using extremely small lenses (such as the camera on a high-end smartphone) - it seems the smaller the lens the bigger the field of view is (this is probably a simplistic way of putting it but hey): using a 8MP smartphone camera gives you the possibility to take images that are sharp all around and capable of taking sharp pictures from very closeby. I believe the PhotoScan manual recommends cameras of 5MP up so I don't see why there's a need for 21MP or the like.

[Kiesel]

VCD, glad you like the article.

To my understanding there are in general two alignment techniques:

A) with only use of the object
B) with the help of the background

to A) You can improve the alignment when you only use the object:

- when you improve lighting for better viewing of the texture/surface structure
- using a higher resolution camera (but 12 MP shouldn'd that bad)
- take closer photographs, but then you get also problems with a smaler DOF, which you can avoid with using a crop camera instead of fullframe or even a good point and shot camera (or even a smartphone as suggested by ThomasVD), a shorter lens, masking out unsharp areas.
- you can also use pattern projection for unstructured/textureless objects, but this is a bit tedious in PhotoScan and no normal workflow.


to B) You can help PhotoScan with the alignment with a background technique for difficult objects where you can't use alone the object itselves. Additional to the suggestion for A:

- a good textured background with a constant relation to your object (for example newspaper, cork, ...)
- textured rings under and above your (vertical, cylindrical) object (bones)
- coded targets around your object
- marker based alignment for aligning separate scans
- use of the "sandbox technique" where you turn your object and your background together (discribed in the "Practicle Guide")

Karsten
 

[Kiesel]

Rather than manually masking the newspaper or marker in each picture, try generating a model of each chunk (at least to the "mesh" stage), then just select the newspaper in the mesh and delete it from the model.
Afterwards to go Tools -> Import -> Import Masks, and choose "from model" in order to mask the newspaper in all the pictures from that chunk.

Or even simpler take a photo before you insert your object in the background and generate the masks from this backgroundphoto.

Karsten

[bigben]

You're getting to test a 21mpx camera...  that would be key element #1. For a lot of stuff I end up scaling the images back to 12-18mpx but for things like this the resolution of fine details is a boon when there is very little tonal detail.

For things like this I'd use a macro lens. For Canon that would be the 50mm or the 100mm IS (the IS model is noticeable sharper than the older model).  For macro work I always choose a magnification to shoot at and then move the camera to change focus.  I'd possibly consider a focusing rail rather than changing the focus of the lens... but it's not that critical until you get around the 1:2 - 1:1 range.

The main thing to keep in mind when you know that everything is not in focus is that you need to be able to produce tie points between the different images. It's easier to remove images than discover you have gaps.  If you can get the vault OK, you could possibly do the skull with just 2 different orientations.

[bigben]

True image stacking isn't an option for me. My job here is to come up with a routine that is easily repeatable and pretty efficient--a method that could be used to digitize an entire collection of vertebrate skeletal remains (or at least the elements deemed most important).

One thing that's missing here is the suggestion of a handheld laser scanner. I work at a large institution and my job is to create workflows for various imaging applications as well. Time how long it takes to capture and process the images for one skull. Multiply that by the size of your collection and calculate the cost in wages for staff to complete this. Then compare that to the time it would take to complete the project with a high quality laser scanner.

[Kiesel]

One thing that's missing here is the suggestion of a handheld laser scanner. I work at a large institution and my job is to create workflows for various imaging applications as well. Time how long it takes to capture and process the images for one skull. Multiply that by the size of your collection and calculate the cost in wages for staff to complete this. Then compare that to the time it would take to complete the project with a high quality laser scanner.

Oh we are way of topic here.   
A laserscanner is indeed acurate and fast, but I think you don't get high quality texture from it. A TOF (time of flight) laserscanner has additional the advantage that it could better scan difficult parts (for example holes) of your object because it don't need a stereo base.
Another option is a structure light scanner, but same problem with texture and not so good with diffcult parts ( because it works with light section/cut).

Back to Photoscan:
I think fastest solution would be:
- turntable/ "sandbox turntable" with coded targets

Your problems with alignment could also come from focusing problems, therefore I would recommend:
- a assisting light for focusing, like a assisting light from a flash light do.
- a tethered solution, focusing in live view (more acurate!) and using the loupe to control the focus on the computer.

Karsten

[DCK]

Undertaking a comparison of PS and laser scanning is precisely what I am up to. One factor is obviously cost. Most natural history institutions are small and resource constrained. So the comparison is between a NextEngine scanner and PS. Based on my calculations, all equipment included, PS is cheaper by a factor of at least 2 and as much as 8. Moreover, a DSLR camera, tripod, basic lighting is already on-hand at almost every natural history institution, so the cost gap is wider in reality.

For bones on which points are easily detected, basically anything that's not white and not both narrow and cylindrical (worst of all, white, narrow and cylindrical), PS is absolutely ingenious. With 65 photos I can capture a complete human skull fantasically. With 80 I am certain to get no holes. On a manual turntable shooting each orientation with the interval timer, I can complete image capture in about 10 minutes. More importantly, I get good results as long as I take decent photos. I don't have to be perfect. If I raise the skull off the base and shoot against a black background, I can complete masking in 15-20 minutes (mask base and bottom edge of skull with rectangular selection, mask the holes on the sides of the skull (temporal fossae) with magic wand). Plus, I can process all images as a single chunk without trouble.

Have a look at the attachment. It's one of the first models I built, a couple of years ago. The image capture is terrible. Uneven lighting. Glare. Shadows. Areas out of focus. Etc. But the model is fantastic. I was hoping things would go this well no matter what specimen I used.

Still working on modeling the difficult specimens based on some of the suggestions in this thread. The results aren't great. I'll report back soon.

[DCK]

Here are the not-so-stellar results. In the solid and shaded views, you can see the skull, the plasticine, and the black wooden surface. In reality, the skull is by far the smoothest of these three surfaces; in the model, the skull is bumpiest.

Camera alignment looks great, however. :/

I think it is simply that the smooth white surface is not easily modeled. This is obvious at every stage from sparse cloud to shaded. I can improve results slightly by gradual selection on the sparse cloud, but not to the level where the model actually resembles the object.

Results are similar with Nikon 12mpx and Canon 21mpx.

Thanks to everyone who's tried to help. Always open to additional suggestions.

[bigben]

Undertaking a comparison of PS and laser scanning is precisely what I am up to. One factor is obviously cost. Most natural history institutions are small and resource constrained. So the comparison is between a NextEngine scanner and PS. Based on my calculations, all equipment included, PS is cheaper by a factor of at least 2 and as much as 8. Moreover, a DSLR camera, tripod, basic lighting is already on-hand at almost every natural history institution, so the cost gap is wider in reality.

We purchased a NextEngine before starting to use PS, and generally for that type of scanner I can get better models more efficiently with PS, especially for larger objects where you have to move the scanner around and manually stitch. We also have issues with the NextEngine software not being able to create a combined surface for large scan sets, requiring surface reconstructions of the multiple meshes via Meshlab.
We're going to be doing some testing with Metronom, comparing laser scanners and photgrammetry and looking at combined workflows, using PS to texture a laser scanned object. One of their laser scanners captures pretty good texture as it also uses the orientation of the surface to determine when to include texture (using imagery when the surface is perpendicular to the scanner).

The problem you have with noise is probably because of the lack of detail in the surface. This wouldn't be a problem with a laser or structured light scanner. With structured light scanners being relatively economical, a combined approach may be feasible, using the strengths of each to make up for weaknesses in the other. That's the motivation behind our upcoming tests anyway... Yet to be tested/ confirmed