Dealing with network jobs that won't cancel

[willfig]

From time to time we get a job on our network that seems to just hang.  There are no errors, and other jobs continue to process.  We can pause and try and abort these jobs from the Network Monitor but they won't cancel.  If you open the project file directly, you get the network processing dialog but also can't cancel from here.  I know one way to cancel the job is to restart the server. But that means losing all the jobs that are queued.  I'm just wondering if there is another way?  Thanks.

Will

[Alexey Pasumansky]

Hello Will,

Is there any node that is assigned to the job? Cancelling the network task sometimes require some clean up step to be performed to remove the temporary data from the project folder and the server cannot do it by itself.

[willfig]

Hi Alexey,

That job was assigned to two nodes.  I paused those nodes and then later actually stopped them completely.  But the job remained in the que.  What's involved in removing the temporary data so the job will fully cancel?

Will

[Alexey Pasumansky]

Hello Will,

Jobs may exist in the queue on the server, even if no nodes are connected to the server. But if some processing has been already applied and the job is cancelled, server creates the "cleanup" task and it will be sent to the available node.

[willfig]

Hi Alexey

I see. So you just have to wait for that cleanup process to finish? The job was still there a few hours after we cancelled it. Finally restarted the server to kill it so we could open the project file again.

Will

[willfig]

We're having this problem again now.  The project was at 99% complete on the dense cloud and then started throwing this error:

2019-03-11 09:10:10 [10.165.18.3:65388] failed #38 BuildDenseCloud.cleanup (1/1): Can't open file: The system cannot find the file specified (2): R:/PRJ-HabitatStr/Processing/Palau/CORALASSIST/Coral_Colonies/PA-DR_A075/PA-DR_A075_2019.02.08.files/0/0/dense_cloud/dense_cloud.zip

It racked up 460,181 of these before we paused (it was over the weekend) and then tried to abort it. Its been sitting paused for two days now.  We can't open the project file as the job is still with the network.

Any help appreciated.

Will