projecting 3D points to image with xml file : only works on the first view

[jungeun]

Hi , I have trouble projecting 3D points in to an raw image with xml format file .
First image's projection looks perfect, but other image's projection looks weird.

To make extrinsic and intrinsic matrix from xml file, I got help from this post "https://www.agisoft.com/forum/index.php?topic=12992.0"
difference with that post is I used component (maybe chunk) transformation too for get extrinsic matrix.

here's my result url. plt figure's range is just normalized to ([0,4000], [0,6000] -> [-1,1] , [-1,1])


https://docs.google.com/presentation/d/19i0ro8IG82ylNYUXxwDFh8J8Or3pAKzMbiPoO4iDtws/edit?usp=sharing


I think this happened because way that i used to make extrinsic  matrix (specifically translation part) is wrong, because after I apply extrinsic matrix from obj file (obj file that meta shape made), it translated from center xy in 3d space too, as you can see from the url.

but now I have no idea to fix extrinsic matrix.
Any help would be very very very appreciated. I used too much time for here.
Thanks for your time !!!

* Here's my way to do projection.

Code: [Select]


 ext_transform = np.matmul(chunk_transform,camera_transform)
 Extrinsic =  np.linalg.inv(ext_transform)
 projection_mat = np.matmul(Intrinsic,Extrinsic)

 with this matrix, do opencv projection



chunk_transform is <component>< transform>'s  [rot | trans]  in XML file.
camera_transform is <camera>< transform>'s  in XML file.
Intrinsic is <sensor> <f>, <cx>, <cy> value in XML file.

here's my xml file and matrix for sure.

Code: [Select]


<components next_id="1" active_id="0">
<component id="0" label="Component 1">
<transform>
<rotation locked="true">-4.0000842670820230e-01 -1.5734843503078316e-01 9.0290349902732769e-01 1.9110407888160198e-02 -9.8637019650377589e-01 -1.6342774476645819e-01 9.1631220163914506e-01 -4.8117620914441349e-02 3.9756338322933804e-01</rotation>
<translation locked="true">3.2904539920717615e+00 -1.8702646000885890e+00 1.2696371691852406e+00</translation>
<scale locked="true">1.2753279589141573e+00</scale>
</transform>
<region>
<center>1.1102797740173276e-01 -1.8681378707747584e-01 -2.8371484315618511e+00</center>
<size>2.3905810713768005e+00 5.7288969755172747e-01 9.4771908521652215e-01</size>
<R>-6.8483242464340593e-02 9.5930334543824825e-01 2.7394732510567710e-01 -3.9456557709561146e-01 -2.7824770616937589e-01 8.7572610979842780e-01 9.1631220163914517e-01 -4.8117620914441078e-02 3.9756338322933699e-01</R>
</region>
<partition>
<camera_ids>0 1 2 3 4 5 6 7 8 9 10 11 12 13</camera_ids>
</partition>
</component>
</components>


<camera id="0" sensor_id="0" component_id="0" label="SSP_L17_1st_">
<transform>9.9989019305885085e-01 3.1326853264383977e-03 1.4484063911040377e-02 5.3838085029389166e-03 2.5733478592101107e-03 -9.9925620803875881e-01 3.8476077980954093e-02 -2.5820096998954243e-02 1.4593824225647072e-02 -3.8434580505665616e-02 -9.9915454426020855e-01 -4.1891819883547572e-02 0 0 0 1</transform>
<rotation_covariance>3.3690760850214915e-05 -2.8655228346104676e-06 2.0765386794765720e-05 -2.8655228346104680e-06 2.0896033454963796e-05 -8.7250725461524285e-06 2.0765386794765720e-05 -8.7250725461524302e-06 8.5295693752518732e-05</rotation_covariance>
<location_covariance>3.0046979288328115e-04 -7.9965393331102712e-05 3.2699651445831571e-05 -7.9965393331102712e-05 2.4640075563414816e-04 -3.3035204858024964e-05 3.2699651445831571e-05 -3.3035204858024964e-05 1.7833428418572984e-04</location_covariance>
<orientation>2</orientation>
</camera>

<camera id="1" sensor_id="1" component_id="0" label="SSP_L17_2nd_2">
<transform>3.2315182958907984e-01 -9.0921069075667355e-02 9.4196934888103856e-01 -2.6283049499955293e+00 -1.4862270882993905e-01 -9.8789816958817889e-01 -4.4367746667785007e-02 1.5221309812608544e-01 9.3460375852726019e-01 -1.2566051775503462e-01 -3.3275403652586294e-01 -1.7708344352327172e+00 0 0 0 1</transform>
<rotation_covariance>6.9188346207251671e-05 3.8657336742568942e-06 -3.3461144497649100e-05 3.8657336742568933e-06 2.0050704462952276e-05 -3.6855288024366330e-06 -3.3461144497649100e-05 -3.6855288024366334e-06 5.0405859802254651e-05</rotation_covariance>
<location_covariance>3.0712499188493430e-04 2.0662592833772486e-04 7.3567010783359092e-05 2.0662592833772486e-04 5.5535682375073474e-04 1.2483174959582446e-04 7.3567010783359092e-05 1.2483174959582446e-04 2.4186295188170692e-04</location_covariance>
<orientation>2</orientation>
</camera>


<sensor id="0" label="Canon EOS 200D II, EF-S18-55mm f/4-5.6 IS STM (24mm)" type="frame">
<resolution width="4000" height="6000"/>
<property name="pixel_width" value="0.0037211000000000002"/>
<property name="pixel_height" value="0.0037211000000000002"/>
<property name="focal_length" value="24"/>
<property name="layer_index" value="0"/>
<bands>
<band label="Red"/>
<band label="Green"/>
<band label="Blue"/>
</bands>
<data_type>uint8</data_type>
<calibration type="frame" class="adjusted">
<resolution width="4000" height="6000"/>
<f>6268.7417135633541</f>
<cx>-42.121378778566942</cx>
<cy>-136.38016191503107</cy>
<k1>-0.096948574201960791</k1>
<k2>0.32612459814959294</k2>
<k3>-0.47617087512721168</k3>
<p1>-0.0016501137249175922</p1>
<p2>-0.0013166070712920016</p2>
</calibration>
<covariance>
<params>f cx cy k1 k2 k3 p1 p2</params>
<coeffs>6.2266898591838116e+00 -1.1817357726592841e+00 -6.7850811513866622e+00 -5.4245864935081574e-03 3.6333477912751252e-02 -6.9388876100023159e-02 -4.6672961693054885e-05 -5.7791029801410488e-05 -1.1817357726592841e+00 1.2623904582211022e+01 6.6555547251095845e-02 9.6480593388115058e-04 -9.9126278634053041e-03 2.4241385273150551e-02 5.8730529886652451e-04 5.6777308739382633e-05 -6.7850811513866622e+00 6.6555547251095845e-02 3.7214137821871702e+01 2.6138982769990197e-05 2.4140826928457713e-03 -8.0175831609379106e-03 2.5761918039468267e-05 5.3870553043905596e-04 -5.4245864935081574e-03 9.6480593388115058e-04 2.6138982769990197e-05 1.3515156156380134e-05 -9.0694676681857923e-05 1.8480680746547185e-04 1.6479371500237986e-08 -1.2786405193345189e-08 3.6333477912751252e-02 -9.9126278634053041e-03 2.4140826928457713e-03 -9.0694676681857923e-05 6.4497209503383188e-04 -1.3645025984899901e-03 -2.4697906723543059e-07 1.3037508646489241e-07 -6.9388876100023159e-02 2.4241385273150551e-02 -8.0175831609379106e-03 1.8480680746547185e-04 -1.3645025984899901e-03 2.9884771129337082e-03 5.5494375728770076e-07 -3.5250720940152735e-07 -4.6672961693054885e-05 5.8730529886652451e-04 2.5761918039468267e-05 1.6479371500237986e-08 -2.4697906723543059e-07 5.5494375728770076e-07 3.1687646647244701e-08 1.4357625948766003e-09 -5.7791029801410488e-05 5.6777308739382633e-05 5.3870553043905596e-04 -1.2786405193345189e-08 1.3037508646489241e-07 -3.5250720940152735e-07 1.4357625948766003e-09 3.1852916864167610e-08</coeffs>
</covariance>
</sensor>

<sensor id="1" label="Canon EOS 200D II, EF-S18-55mm f/4-5.6 IS STM (25mm)" type="frame">
<resolution width="4000" height="6000"/>
<property name="pixel_width" value="0.0037211000000000002"/>
<property name="pixel_height" value="0.0037211000000000002"/>
<property name="focal_length" value="25"/>
<property name="layer_index" value="0"/>
<bands>
<band label="Red"/>
<band label="Green"/>
<band label="Blue"/>
</bands>
<data_type>uint8</data_type>
<calibration type="frame" class="adjusted">
<resolution width="4000" height="6000"/>
<f>6736.4734574874947</f>
<cx>-47.408453741129755</cx>
<cy>-159.8371889525549</cy>
<k1>-0.10092234606082934</k1>
<k2>0.44971968975065907</k2>
<k3>-0.56200764088241306</k3>
<p1>-0.0019806263234060204</p1>
<p2>-0.00012688410818793622</p2>
</calibration>
<covariance>
<params>f cx cy k1 k2 k3 p1 p2</params>
<coeffs>5.4956621533466237e+00 -6.0038039568943236e-01 -6.5544840324940603e+00 -4.3031618281647345e-03 3.2245294427993405e-02 -6.5399303962367927e-02 -2.9948025414546900e-05 -9.2822328790459729e-05 -6.0038039568943236e-01 1.1688816590997790e+01 -7.4820546554811573e-01 9.2918626142461354e-04 -7.3752261208181206e-03 1.3712045369149686e-02 5.7392942778724317e-04 -9.8327211602412058e-06 -6.5544840324940603e+00 -7.4820546554811573e-01 4.3617799922841485e+01 -1.8409220364205559e-03 2.1705650370708315e-02 -5.2182603024796409e-02 -3.0348125806527517e-05 6.3564045538338034e-04 -4.3031618281647345e-03 9.2918626142461354e-04 -1.8409220364205559e-03 1.4202361019538604e-05 -1.1064952256554250e-04 2.6627173737172130e-04 4.1306591286570337e-08 -3.1831830072314878e-08 3.2245294427993405e-02 -7.3752261208181206e-03 2.1705650370708315e-02 -1.1064952256554250e-04 9.2198360214500944e-04 -2.3126132536480113e-03 -3.4261105953989335e-07 3.7828580014534883e-07 -6.5399303962367927e-02 1.3712045369149686e-02 -5.2182603024796409e-02 2.6627173737172130e-04 -2.3126132536480113e-03 6.0581666948223692e-03 6.6263904122748835e-07 -9.6030861828633363e-07 -2.9948025414546900e-05 5.7392942778724317e-04 -3.0348125806527517e-05 4.1306591286570337e-08 -3.4261105953989335e-07 6.6263904122748835e-07 3.1086531077208177e-08 -1.1551649144677374e-09 -9.2822328790459729e-05 -9.8327211602412058e-06 6.3564045538338034e-04 -3.1831830072314878e-08 3.7828580014534883e-07 -9.6030861828633363e-07 -1.1551649144677374e-09 3.5706106706884163e-08</coeffs>
</covariance>
</sensor>

for detail and confirm, I attach matrix i made with that xml file

1. First Image's matrix

Code: [Select]

chunk_transform = np.matrix([[-4.0000842670820230e-01, -1.5734843503078316e-01 , 9.0290349902732769e-01, 3.2904539920717615e+00] ,
                   [1.9110407888160198e-02, -9.8637019650377589e-01, -1.6342774476645819e-01, -1.8702646000885890e+00] ,
                   [9.1631220163914506e-01 ,-4.8117620914441349e-02, 3.9756338322933804e-01, 1.2696371691852406e+00],
                   [0,0,0,1]])

Intrinsic = np.matrix([[6312.3568293358312, 0, 2000+105.50729729807456], [0, 6312.3568293358312, 3000-183.65292103661881], [0, 0, 1]])
Extrinsic = np.matrix([[ 9.9989019305885085e-01, 3.1326853264383977e-03, 1.4484063911040377e-02, 5.3838085029389166e-03 ],
                       [ 2.5733478592101107e-03, -9.9925620803875881e-01, 3.8476077980954093e-02, -2.5820096998954243e-02 ],
                       [ 1.4593824225647072e-02, -3.8434580505665616e-02, -9.9915454426020855e-01, -4.1891819883547572e-02 ],
                       [ 0,0,0,1 ]])



2. second Image's matrix

Code: [Select]

chunk_transform = np.matrix([[-4.0000842670820230e-01, -1.5734843503078316e-01 , 9.0290349902732769e-01, 3.2904539920717615e+00] ,
                   [1.9110407888160198e-02, -9.8637019650377589e-01, -1.6342774476645819e-01, -1.8702646000885890e+00] ,
                   [9.1631220163914506e-01 ,-4.8117620914441349e-02, 3.9756338322933804e-01, 1.2696371691852406e+00],
                   [0,0,0,1]])

Intrinsic = np.matrix([[6733.6864664607137, 0, 2000+111.00756885316275], [0, 6733.6864664607137, 3000-228.02990467572579], [0, 0, 1]])
Extrinsic = np.matrix([[3.2315182958907984e-01 ,-9.0921069075667355e-02 ,9.4196934888103856e-01, -2.6283049499955293e+00 ],
                       [-1.4862270882993905e-01 ,-9.8789816958817889e-01 ,-4.4367746667785007e-02 ,1.5221309812608544e-01 ],
                       [9.3460375852726019e-01 ,-1.2566051775503462e-01 ,-3.3275403652586294e-01 ,-1.7708344352327172e+00 ],
                       [0, 0, 0, 1]])


Thank you for your time again.

[Paulo]

Hello,

I did the recalculation of chunk_transform, camera_transform as well as Extrinsic. In case of first image Iget:

Code: [Select]

camera_transform # first image
Out[62]: 2022-05-02 00:18:20
2022-05-02 00:18:20 array([[ 0.99989019,  0.00313269,  0.01448406,  0.00538381],
2022-05-02 00:18:20        [ 0.00257335, -0.99925621,  0.03847608, -0.0258201 ],
2022-05-02 00:18:20        [ 0.01459382, -0.03843458, -0.99915454, -0.04189182],
2022-05-02 00:18:20        [ 0.        ,  0.        ,  0.        ,  1.        ]])

chunk_transform
Out[63]: 2022-05-02 00:18:23
2022-05-02 00:18:23 array([[-0.51014193, -0.20067086,  1.15149808,  3.29045399],
2022-05-02 00:18:23        [ 0.02437204, -1.25794549, -0.20842397, -1.8702646 ],
2022-05-02 00:18:23        [ 1.16859857, -0.06136575,  0.5070237 ,  1.26963717],
2022-05-02 00:18:23        [ 0.        ,  0.        ,  0.        ,  1.        ]])

ext_transform = np.matmul(chunk_transform,camera_transform)

Extrinsic =  np.linalg.inv(ext_transform)

Extrinsic
Out[66]: 2022-05-02 00:18:48
2022-05-02 00:18:48 array([[-0.30360238,  0.01112263,  0.72286483,  0.09731044],
2022-05-02 00:18:48        [ 0.09509364,  0.77782164,  0.02797098,  1.07889018],
2022-05-02 00:18:48        [-0.71666902,  0.09849601, -0.30251569,  2.88552426],
2022-05-02 00:18:48        [ 0.        ,  0.        ,  0.        ,  1.        ]])
The difference for chunk_transform is that it is formed by  [s*rot | trans] and not  [rot | trans] where s is scale ....

also you cannot do matrix multiplication of Intrinsic and Extrinsic as they do not have same order (3,3) (4,4)

Once you get camera coordinates (Xcam, Ycam, Zcam) from applying Extrinsic to World or Model coordinates you must transform to homogeneous coordinates (Xcam/Zcam, Ycam/Zcam, 1) before applying Intrinsic matrix to get pixel coordinates. Note Intrinsic does not take into account distortion (k1,k2,k3,p1,p2)...

Hopes this helps you get along the way to more wisdom...

[jungeun]

It perfectly works after I change chunk matrix to [s*R | T] instead [R|T] .
I would never have thought of multiplying s to only R. Really appreciate it.

In the case of the dimension of the array, it was processed separately although it was omitted in the article. 
Thank you very much!!!!!!!!