Caution: Long post, more to clarify my own thoughts that applying directly to your kelp problem, and I'm really tired so please excuse any typos.
I'm completely new to this too (just got the demo to make a decent mesh *today*), and I'm having a similar problem.
I put a small bust on a turntable and shot a series of photos. Photoscan interpreted the camera positions as being all piled up and the result was not even vaguely related to the original object. Tried several times, all fails.
Conclusion: Need to learn more about suitable photography, need an object with more obvious surface points for the software to match.
Action: Watched a few tutorials. Need flatter lighting, more obvious details for the software to match up.
Today I went outside and shot a number of small areas in open shade according to the instructions. Loaded them in and the cameras/viewpoints dispersed properly, and the result from several was a pretty decent rendition of the area, including the uncolored mesh. Not terrible for ten photos.
Conclusion: Yes, I need to work on understanding how to produce photos acceptable to the software.
Action: Loaded up the several other image sets and produced reasonable meshes considering the small number of photos. Conclusion verified.
Loaded up a short run of images made on a turntable shortly after the first one. While there were some apparently excessively smooth/featureless areas, and the actual mesh had a low accuracy-of-feature level, the software did actually disperse the cameras properly and only meshed the areas that had sufficient details to match between images.
Conclusion: Supports other conclusions. Increase resolution/polygon settings.
I took a small painted figurine and set up the tripod, turning the figurine slightly between shots. Software stacked many of the cameras and had a difficult time matching the surface points, but I *did* end up with a gummy-looking side of the object. It's really bad, but it's quite a bit better than what I had *yesterday*.
Conclusion: Need to take sharper macro photos, need to use a method for eliminating the background, support structures, and etc.
Action: I'll have to make some things for that. Need a black or green screen background to make masking fast, and softbox style lighting - or a thing with which to swing the camera around in an orbit. Large white areas are a problem.
To get to your kelp problem, is the kelp shiny? That's a bad thing, can you dull it up with talc/cornstarch/ nondairy creamer/flour/dry deoderant spray (an old standby for photographing shiny spoons), or something else?. Is there other stuff in the image that doesn't move while the kelp does? That seems to be one of the sources of camera piling, at least in my case.
There's things in some of the tutorials about using a plain background shot as a semi-auto mask.
It seems like the lack of contrast you mentioned might also be a problem, I'm not sure how to approach that. Now that I'm registered and can see your photo, I can see that that will probably be a very difficult subject. It might be easier to do thin slices and then group them together in a modeler.
One of the next steps in my project direction is figuring out how to get it to handle photographs of a small sculpt in a smooth white material - which seems to be one of the worst for this. Since I can deface the thing, while needing the result to to be as smooth as possible, I plan on starting by covering a small test area with pencil marks, whether dispersed dots or a grid pattern of some sort. I can go with different forms of "structured light" if I have to, it's just a matter of dragging things around and setting it up ( as if I don't already have more to do than I can accomplish)
Sure, I could spend about six hundred bucks and have a shop do the scan and clean it up for a rescaled output, but I'm on a pocket-change budget and besides, I still need to figure out how to get from the proposed objects to a viable model. If I can do one, I'll need to do about six more, and there will be more in the future.
And, we have a symmetry problem, so if I can get a good model of one side of the thing, I can mirror it and have it output at a cost far less than the two weeks that have already been applied to not getting it right. I'm even working on a method of doing that mechanically with stuff I already mostly have lying around.
I'm still working/testing in demo mode, but if I can arrive at a process that works then the software will pay for itself - at least the basic version.
Time for me to go try to sleep. I'll be fighting this in my dreams, in a half-meshed world, always with some sort of giant spiders of some sort - the two pounders aren't a problem, it's the 300+ pounders...
Good luck with the project,
-Gary