Network processing Error: Can't remove folder: parent folder mismatch

[jrp]

Network processing Error: Can't remove folder: parent folder mismatch

I’ve just updated everything to 2.1.0 and was excited to get network processing back up and running (having disbanded the setup during covid). Setup is as follows:

Clients run windows.
Clients have basic bat script on the desktop so that users can run underused clients as processing nodes.
We are running a pair of new linux servers as processing nodes, though we used to have an old linux server running as a processing node too sucsessfully.
The “server” is the same physical machine as the file server, which is a linux machine with samba and zfs hosting the data.
(there is a quirk in the current setup in that as part of the new server deployment process, windows machines are temporarily connecting into an old linux server with samba, and nfs is mapping the shared folders on the actual new server.)

Prior to the new server installs, I had this all working fine, and I have a fairly good understanding of how it’s supposed to work.

I connected everything, ran a windows client using it’s bat script, created a new project using the same machine, then tested network processing.

Image alignment worked great. Then I did model from depth maps, it almost finished then it repeatedly gave:
failed #2 BuildModel.cleanup (1/1): Can't remove folder: parent folder mismatch

This was running on one of the linux nodes. When I used node priorities to shift the job to the windows node, it completed fine soon after.

It’s clear that the paths are set (at least mostly) right as the linux nodes did most of the processing just fine.

Any ideas on what’s going on here?

Console output on the node is (repeated every few seconds forever):

2024-01-05 15:18:34 processing failed in 0.211594 sec
2024-01-05 15:18:35 BuildModel.cleanup (1/1): subtask = cleanup, surface_type = Arbitrary, interpolation = Enabled, face_count = High, source_data = Depth maps, vertex_colors = on, volumetric_masks = off, blocks_crs = Local Coordinates (m), working_folder = /home/saceimg/photogrammetry/test2/test/test1a1.files/0/0/model.tmp
2024-01-05 15:18:35 Peak memory used: 4.80 GB at 2024-01-05 15:15:48
2024-01-05 15:18:35 deleting all temporary files...
2024-01-05 15:18:35 Error: Can't remove folder: parent folder mismatch

I suspected a permissions issue as there are different users in use, but I tried waiting until it crashed, then using a linux node, logged in as the correct user, deleted all of the data in the folder mentioned, and no permissions showed. Interestingly with the model.tmp folder deleted, it continued with the same error message, but was fine when the job was passed back to windows.

Thanks,
JRP

[jrp]

Update (and sorry for the double post):

I’ve managed to get it to show the issue the other way round, where, (by using priorities again) I start processing the job on the linux node, then transfer it to the windows node, the windows machine gives the message (with the server name removed)

2024-01-08 10:01:03 BuildModel.prepairDepthMaps (3/4): subtask = prepairDepthMaps, surface_type = Arbitrary, interpolation = Enabled, face_count = Low, source_data = Depth maps, vertex_colors = on, volumetric_masks = off, blocks_crs = Local Coordinates (m), working_folder = //[snip].uk/groups/photogrammetry/test2/test/test1a1.files/0/0/model.tmp
2024-01-08 10:01:03 Error: Can't open file: The network name cannot be found (67): //[snip].uk/groups/photogrammetry/test2/test/test1a1.files/0/0/model.tmp/depth_maps_pyramids/partitions.groups

Interestingly, the /groups/… bit is wrong, and Metashape should theoretically not even know that a folder named “groups” exists, but that is the actual location that the data exists at on the server. This is hidden from windows at the samba level, so the windows machine cannot be the source of the issue. On Windows Metashape is called with:

C:\Program Files\Agisoft\Metashape Pro>"C:\Program Files\agisoft\metashape pro\metashape.exe" --node --dispatch [snip].uk --root \\[snip].uk\jrp

Samba mounts \\[snip].uk\jrp to /home/jrp/ on the server

On Linux, Metashape is called as:

/opt/metashape-pro/metashape.sh --worker --host [snip].uk -platform offscreen --root /home/saceimg/

And both user accounts have symlinks to the actual data:
/home/jrp/photogrammetry > /groups/photogrammetry/
/home/saceimg/photogrammetry > /groups/photogrammetry/

On the Linux processing nodes, autofs and nfs is used to get the /groups/photogrammetry folder on the node to point at the /groups/photogrammetry folder on the fileserver.

My current theory is that this is a Metashape bug: the Linux node is somehow resolving the full path of the files it’s working on, instead of using the path via the symlink as I had expected it should. It’s then passing that full path back to the windows node, that can’t understand it. Though I’m not sure that explains everything.

Do remember that processing works part way, so this is not a simple path issue.

I suspect I can work around the issue using either bind mounts, or adding a second symlink for windows to use to the “groups” folder if I have to.  But those are both far from ideal given the setup here.

I’m 90% sure that this configuration worked fine with mixed Windows/Linux processing nodes last time we tried it a few years back. It would be difficult for me to confirm it was actually a symlink I used, but I can’t think of any other way I could have done it.

Thanks,
JRP

[Alexey Pasumansky]

Hello JRP,

Are you able to to start from scratch network processing on Windows node only, using the give root path?

Can you also check, whether the root path is properly defined in the Network Preferences tab of the client application (where from the processing is started) and that "Absolute paths" option is disabled?

[jrp]

Hi,

I have tried networking processing on linux nodes only which completes good.

I have also now tried processing on the windows node only, and this completes fine as well.

(I used the pause node option in network monitor)

Only when it’s a mix of linux and windows nodes does the problem show up. This is consistent with my idea that the Linux client is resolving the symlinks and passing the resolved versions to windows, which can’t find them.

Path is correct and absolute paths is off.

Thanks,
JRP