Dense Cloud Generation + Network Processing = Stuck at 5%

[jenkinsm]

Hi all,

I am trying out network processing for the first time to see if I can speed up my very large projects. Normally I generate a mesh from depth maps, but I discovered that making a point cloud first gives me the ability to remove trees, cars, and houses before making a mesh (these look bad in the final product and removing from the mesh has proven to be difficult and laborious).

My current project has over 22000 35 MP images in EXR format. The photos are on two drives, but both drives are shared on the network and both nodes can access all the photos. The depth maps are already created in Medium quality, and I am reusing  the depth maps to generate a point cloud.

Currently, the Network Monitor shows the progress is at 5% and it hasn't progressed in the last hour.

Should I continue to let this run or do I need to change something in order for this to work?

I do not have any tweaks enabled and the images cover a 5 km road, so each photo has probably 15-20 neighbors max.

Also, the monitor shows this running on 1/1 node, so I'm wondering if it's still figuring out how to split up the job.

The CPU on the active node is at 100% but it's barely using any memory or disk.. So it's doing something, but I don't know what!

Any advice is appreciated. Thanks!

This is what the log states currently:

2022-12-13 14:51:36 BuildDepthMaps: quality = Medium, depth filtering = Aggressive, PM version, reuse depth maps
2022-12-13 14:51:37 processing finished in 1.583 sec
2022-12-13 14:51:38 BuildDenseCloud: point colors = 1
2022-12-13 14:51:40 Generating dense point cloud...
2022-12-13 14:51:40 processing finished in 1.587 sec
2022-12-13 14:51:41 BuildDenseCloud.initialize (1/1): point colors = 1
2022-12-13 14:51:42 Generating dense point cloud...
2022-12-13 14:51:42 initializing...
2022-12-13 14:53:18 selected 22774 cameras in 95.725 sec
2022-12-13 14:53:18 working volume: 1874191x558706x299686
2022-12-13 14:53:18 tiles: 458x136x73

[jenkinsm]

It took over 2 hours but eventually it moved past 5%. I'll leave this up in case others have a similar problem. I believe it was spending that entire time determining how to divide up the task.

[Alexey Pasumansky]

Hello jenkinsm,

How does the area covered by the images look like in respect of the bounding box? Also if you haven't cancelled the task, please provide later the information about the number of sub-tasks that is the dense cloud generation procedure split into.

Also I have a side question regarding the source data sharing - are the both network drives that you have mentioned accessible through the same root path?

[Echostorm]

Yes, I have the same problem as this. I deal with many more photos though. We run corridor surveys with 100,000+ images. We also network process, however 1 node gets stuck for quite some time at 5% burning only CPU. It gets through it eventually however sometimes it can sit there for days while it proceeds. Makes the processing computer that is running on unusable due to CPU load which hovers at ~70% metashape dedication.

[Alexey Pasumansky]

Hello Echostorm,

What settings for the image matching/alignment do you use?

Most likely the problem that you are facing is related to the big number of images being aligned in the same chunk. Usually we recommend to split large projects for alignment purposes by 70,000 - 90,000 images (numbers may  vary depending on image resolution, color depth, reference information and etc.).