How long should reconstruction take/how to tell if job is hanging

[JamesSz]

Hello,

I'm trying to do a relatively large reconstruction (~3500 images) via the Python API, and the job is taking a really long time (~7 days as of now). I have a GPU and 128GB of RAM so I'm a little surprised that it's taking so long.

I'm trying to figure out at what stage of the reconstruction the script is right now, but I am having trouble deciphering the log files. It seems like the job has stalled at the optimize cameras step, but I can't tell if it's still making progress or if it has entered some sort of loop.

I've copied my script and the last 30 lines of the console logs below, and have attached a copy of the full console logs. I'd appreciate any insights.

Thanks!

Script:
    Metashape.app.gpu_mask = 2 ** (len(Metashape.app.enumGPUDevices())) - 1
    doc = Metashape.Document()
    chunk = doc.addChunk()
    photos = glob.glob('/extvol/images/*.png')
    chunk.addPhotos(photos)
    chunk.matchPhotos(generic_preselection=True, reference_preselection=False)
    chunk.alignCameras()   
    chunk.optimizeCameras()
    chunk.buildDepthMaps(downscale=4, filter_mode=Metashape.MildFiltering)
    chunk.buildDenseCloud()
    chunk.buildModel(surface_type=Metashape.Arbitrary, face_count=Metashape.HighFaceCount, interpolation=Metashape.EnabledInterpolation, vertex_colors=True)
    chunk.buildUV()
    chunk.buildTexture()
    chunk.buildOrthomosaic(resolution_x=0.0701037, resolution_y=0.0701037)
    doc.save('result.psz')

Logs (more attached, couldn't upload full file as it was too large):
...
pair 1224 and 1233: 0(0) robust from 186
pair 1217 and 1221: 0(0) robust from 185
pair 1213 and 1226: 0(0) robust from 188
pair 1236 and 1239: 0(0) robust from 188
adding photos 1206 and 1210 (202 robust)
adding 204 points, 0 far (2 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxxx 0.293135 -> 0.293113
adding 0 points, 0 far (2 threshold), 0 inaccurate, 0 invisible, 0 weak
optimized in 312.501 seconds
adding camera 1208 (3 of 34), 142 of 144 used
adding camera 1209 (4 of 34), 140 of 142 used
adding camera 1212 (5 of 34), 138 of 138 used
adding camera 1207 (6 of 34), 136 of 137 used
adding camera 1214 (7 of 34), 135 of 136 used
adding camera 1211 (8 of 34), 129 of 130 used
adding camera 1217 (9 of 34), 123 of 123 used
adding camera 1213 (10 of 34), 113 of 115 used
adding camera 1216 (11 of 34), 99 of 102 used
adding camera 1218 (12 of 34), 96 of 98 used
adding 3024 points, 5 far (2 threshold), 0 inaccurate, 2 invisible, 0 weak
adjusting: xxxxxxxxxx 0.281326 -> 0.278202
adding 3 points, 1 far (2 threshold), 1 inaccurate, 2 invisible, 0 weak
optimized in 212.6 seconds
adding camera 1219 (13 of 34), 462 of 468 used
adding camera 1221 (14 of 34), 420 of 428 used
adding camera 1215 (15 of 34), 414 of 421 used
adding camera 1222 (16 of 34), 354 of 357 used
adding camera 1224 (17 of 34), 323 of 324 used
adding camera 1223 (18 of 34), 303 of 304 used
adding camera 1226 (19 of 34), 300 of 310 used
adding camera 1220 (20 of 34), 289 of 293 used
adding 1953 points, 22 far (2 threshold), 0 inaccurate, 2 invisible, 0 weak

[Alexey Pasumansky]

Hello JamesSz,

It seems to be the alignment stage.

Have you checked, if there's no problem with the available RAM?


I can recommend to save the project in PSX format in the very beginning (right after creating "doc" variable) and also save the project using doc.save() after each processing operation.
In addition to simple backup of the processing results with PSX format you would also allow for the fine-level task subdivision (would help, if you are reaching RAM limits with the current approach).
Additionally you cannot generate or save Orthomosaic, DEM or Tiled Model in PSZ project format (or in the untitled, i.e. unsaved project).

You may also need to use resetRegion method after optimizing the alignment. And for the mesh generation I would recommend to use the depth maps source option, as for 3500 images the dense cloud based reconstruction may be too resource demanding, even for Medium quality.