Object coordinates through image projections

[FabianN]

Hi all,

I'd like to get 3D object coordinates (Metashape coordinates are fine) from a set of image coordinates (2D). As far as I can see, there are three options:

1) Use the 'pickPoint' function of the 'model' (or 'denseCloud') class. With 'cam' being on of the cameras in the chunk and image x,y coordinates as 'img_x', img_y':

Code: [Select]

pt_2d = Metashape.Vector([img_x, img_y])
pt_3d = chunk.model.pickPoint(cam.center, cam.unproject(pt_2d))

2) We can use the depth map of the camera:

Code: [Select]

depth = chunk.depth_maps[cam].image()
depth_val = depth[img_y, img_x][0]
pt_2d  = Metashape.Vector([img_x,img_y])
img_3d = cam.unproject(pt_2d)
ray = img_3d - cam.center
ray /= ray.norm()
pt_3d  = cam.center + depth_val* ray

3) Or we can render a depth image from the model and use it as in 2).

I prefere to use either method 2) or 3) as it is considerably faster for a lot of points (the depth map has to be extracted only once for one camera)
The results from 3) and 1) should be identical but they are not. Multiple points on a flat surface are also flat in 3D by method 1). For method 2) or 3) however, they are not.

Questions:
#1: Are the rendered depth images / maps corrected for image distortions or do I have to undistort them?
#2: What is missing in method 2)/3) to get the correct results?

Thank you.

[Paulo]

Hello FabianN,

the depth maps are in undistorted image coordinates. So for point pt_2d with img_x, img_y pixel coordinates in camera, its corresponding image coordinates on depth map are:

Code: [Select]

img_x_un, img_y_un = dm.calibration.project(cam.calibration.unproject(pt_2d))
where dm = chunk.depth_maps[cam]
so now to calculate the 3d coordinates using depth map, you would do:

Code: [Select]

depth = chunk.depth_maps[cam].image()
depth_val = depth[int(img_x_un), int(img_y_un)][0]
pt_2d  = Metashape.Vector([img_x,img_y])
img_3d = cam.unproject(pt_2d)
ray = img_3d - cam.center
ray /= ray.norm()
pt_3d  = cam.center + (depth_val/cam.transform.inv().mulv(ray).z)* ray
where depth_val is scaled along camera Z axis (viewing direction)

Values with methods 1 and 2 should correspond much better

[FabianN]

Hi Paulo,

Thanks for your answer. Indeed the results now look very good.
I was thinking about scaling the depth value by dividing the cosine of the angle between the camera principal point and the actual image coordinate. Which turns out is equivalent to your expression 'cam.tranform.inv().mulv(ray).z' (that is much more elegant and faster). Could you explain your expression in more detail so that its geometrical meaning is more obvious?

Thank you and best regards,
Fabian
 

[Paulo]

Hey FabianN,

your normalized ray vector is expressed in internal CS. But the depth value is the distance from camera center in camera viewing direction, i.e. the Z component for the local camera CS where X points to right of image, Y down and Z points in viewing direction. So to correctly scale the ray vector we must transform it to camera local CS to get its z component (cam.transform.inv().mulv(ray).z) to use as basis for scaling relative to depth_val (depth now  corresponds to Z component of ray). That is what cam.trasform.inv() does, transform from internal CS to local camera CS...

Hope this makes sense,

[FabianN]

Hi Paul,

Yes it is clear now. Thanks for the clarification.

Cheers