Camera coordinates to world coordinates using 4x4 matrix

[callahb]

I want to convert camera coordinates to world coordinates.

I have values in the camera coordinate system as X Y Z and I have the 4x4 matrix from the Photoscan XML camera file. Do I just multiply the 4x4 matrix (M) by the camera vector 4x1 matrix (Pc) to yield a world coordinate or is there more to do?

[Alexey Pasumansky]

Hello callahb,

chunk.transform * vector(x, y, z, 1) will give you coordinates in the geocentric coordinate system. To get coordinates in the projected/geographic coordinate system you also need to use the coordinate system info.

Via Python it will look like:

Code: [Select]

vector = PhotoScan.Vector([x, y, z])
coord_geoc = chunk.transform.matrix.mulp(vector)
coord_proj = chunk.crs.project(coord_geoc)

[callahb]

Thank you, that looks like it would pretty easy using the Photoscan python tools.

I'd like to understand the formulas behind the functions. I just found an older post you'd written about this and you wrote the formula as:

"transform vector from camera coordinate system into world's coordinates. M x Vc = Vw (M - camera rotation matrix, Vc - vector in camera's coordinates, Vw - vector in world's coordinates)"

Based on your message, is the Vw in the geocentric coordinate system if I used M from the XML file? What is the formula to convert Vw to a projected coordinate system?

[Alexey Pasumansky]

Hello callahb,

The chunk transformation (4x4) matrix in XML is stored in the <transform> section in the very end of <chunk> section (don't mix it with the individual transformation matrices for each camera). When this matrix is being multiplied on the extended (x,y,z,1) vector PhotoScan is calculating the position of the same point in the geocentric coordinates (but in case for the chunk the Local Coordinates option is selected, it will already give you the local coordinates as a result).

[callahb]

OK Thank you.

I'm still a little confused after reviewing the values in the XML <transform> and the

Code: [Select]

Chunk.transform.matrix
because the values are different.  Here are the results:

-<transform>
<rotation>1.2920001487202448e-001 6.1813988403410725e-001 7.7537761118268644e-001 9.8298306585324424e-001 -1.8280649955923373e-001 -1.8057573609335055e-002 1.3058196048009565e-001 7.6451610021320004e-001 -6.3123964079577122e-001</rotation>
<translation>-2.4914770625398587e+006 -3.7988877821658887e+006 4.4618952101149326e+006</translation>
<scale>1.4135476110541292e+000</scale>
</transform>

>>> print (PhotoScan.app.document.chunk.transform.matrix)
Matrix([[0.18263037237050816, 0.8737701563736887, 1.096033169952144, -2491477.0625398587],      [1.3894933644435172, -0.25840569073712255, -0.025525240036909657, -3798887.7821658887],  [0.18458381828340392, 1.0806799070687882, -0.892287286249529, 4461895.210114933], [0.0, 0.0, 0.0, 1.0]])

[Alexey Pasumansky]

Hello callahb,

You need to multiply Rotation 3x3 matrix by scale, before using it along with the translation vector for the 4x4 matrix creation.

[callahb]

That's it! Thank You!

[callahb]

Alexey,

I've been having difficulty getting a decent result when doing the conversion from local camera coordinates to CRS coordinates. I create X,Y,Z in a camera local coordinate system from [u,v,depth]. So the coord system X is to the right, Y is down, and Z along the optical axis.

Code: [Select]

cam_vector = PhotoScan.Vector([cam_X, cam_Y, cam_Z]) #camera local coordinate system
cam_matrix = camera.transform
chunk_matrix = chunk.transform.matrix
coord_internal = cam_matrix.mulp(cam_vector)
coord_geoc = chunk.transform.matrix.mulp(coord_internal)
coord_proj = chunk.crs.project(coord_geoc)

If I use this method starting with the pixel x,y from a Marker (a well known point), the ending Marker X,Y,Z in CRS coords is typically very far off. Can you help?

[Alexey Pasumansky]

Hello callahb,

Which task you are trying to solve?

[callahb]

I am working on an external application/process where I present the user a 2D oblique image. They click on the image and a 3D projected coordinate is generated for the pixel location of the click.

I am using Photoscan to align all of the photos and thus create the necessary transforms to do the pixel to 3D coordinate conversion and to create the depth images as well.

[Alexey Pasumansky]

Hello callahb,

Using the point projection on 2D photo you can only get the ray though this point and camera center, not 3D point. But you also need to utilize the calibration information, that you are not using in your code.
Also the units of the camera coordinate system are not pixels.

Code: [Select]

cam_matrix = camera.transform
cam_calib = camera.sensor.calibration
chunk_matrix = chunk.transform.matrix
point_2D = PhotoScan.Vector([image_x, image_y])
vect = cam_calib.unproject(point_2D)
ray = cam_matrix.mulv(vect)
Having this ray you can get the position of 3D point on it, as you can get the camera.center (on the same ray) and, as I understood, you have also the distance from the camera to the point (depth).

[callahb]

Hi Alexey,

I've made a little progress but I seem to be stuck again. So far I have the following:

Code: [Select]

cam_matrix = camera.transform
cam_calib = camera.sensor.calibration
chunk_matrix = chunk.transform.matrix
point_2D = PhotoScan.Vector([image_x, image_y])
vect = cam_calib.unproject(point_2D) #vector in camera crs
ray = cam_matrix.mulv(vect) #ray in chunk crs

#find vector/ray intersection with depth plane in front of camera, not sure if this is correct
t = (depth-camera.center[2])/ray[2] #determine parameter t
point_chunk = camera.center - t * ray #find point

point_chunk = PhotoScan.Vector([point_chunk_x,point_chunk_y,point_chunk_z])
point_geoc = chunk_matrix.mulp(point_chunk)
point_crs = chunk.crs.project(point_geoc)


When I test the above with an existing Marker where I already know the [image_x, image_y] and the XYZ in the project CRS, the values don't match. Can you tell me where I've gone wrong?

[Alexey Pasumansky]

Hello callahb,

How to you get depth?

Also not sure if it's a typo or not, but there are no point_chunk_x, point_chunk_y and point_chunk_z variables.

[callahb]

Yes, it was a typo, sorry. Here's the full code:

Code: [Select]

#Compare Marker Coords Calculated from Depth Versus Estimated From Model Alignment
import PhotoScan

chunk = PhotoScan.app.document.chunk #active chunk
scale = chunk.transform.scale
camera = chunk.cameras[0] #first camera in the chunk
depth = chunk.model.renderDepth(camera.transform, camera.sensor.calibration) #unscaled depth
depth_scaled = PhotoScan.Image(depth.width, depth.height, " ", "F32")
v_min = 10E10
v_max = -10E10

#create array of distance values
for y in range(depth.height):
    for x in range(depth.width):
        depth_scaled[x,y] = (depth[x,y][0] * scale, )
       
marker = chunk.markers[0] #first marker
x0,y0 = marker.projections[camera].coord

image_x = int(x0) #marker pixel coords in first camera
image_y = int(y0)

depth = depth_scaled[image_x,image_y][0] #marker distance from camera
focal_length = chunk.sensors[0].calibration.f #camera focal length

cX = chunk.sensors[0].calibration.width/2 #image center x
cY = chunk.sensors[0].calibration.height/2 #image center y

u = image_x - cX #col offset from image center
v = image_y - cY #row offset from image center

cam_X = u*depth/focal_length #camera coordinates X
cam_Y = v*depth/focal_length #camera coordinates Y

cam_matrix = camera.transform
cam_calib = camera.sensor.calibration
chunk_matrix = chunk.transform.matrix
point_2D = PhotoScan.Vector([image_x, image_y])
vect = cam_calib.unproject(point_2D) #vector in camera crs
ray = cam_matrix.mulv(vect) #ray in chunk crs

#find vector/ray intersection with depth plane in front of camera, not sure if this is correct
t = (depth-camera.center[2])/ray[2] #determine parameter t
point_chunk = camera.center - t * ray #find point

point_geoc = chunk_matrix.mulp(point_chunk)
point_crs = chunk.crs.project(point_geoc)
print ('point_crs: ',point_crs)

marker_est = chunk.crs.project(chunk_matrix.mulp(marker.position)) #compute projected coordinates (estimated)
print ('Estimated marker crs coords: ',marker_est)


[callahb]

Hi Alexey,

Have you had a chance to think about this issue yet? Thanks!