Forum

Author Topic: Python API run batch file (start process on cluster)  (Read 7510 times)

Kisioj

  • Newbie
  • *
  • Posts: 5
    • View Profile
Python API run batch file (start process on cluster)
« on: February 16, 2016, 05:08:52 PM »
Hello,

1. Is it possible to run batch file using your Python API? If the answer is yes, then how to do it? Python API documentation doesn't have "cluster", "network" or "batch" words in it.


2. When I select Align Photos from Workflow menu I am asked if I want to schedule processing over network but when I use Python API function "PhotoScan.app.document.chunk.matchPhotos" and "PhotoScan.app.document.chunk.alignCameras" I am not asked about it and the process is started locally. Is it possible to use those API functions to run process over network?

Thanks



Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14927
    • View Profile
Re: Python API run batch file (start process on cluster)
« Reply #1 on: February 16, 2016, 05:17:44 PM »
Hello Kisioj,

To run the batch using scripting, you need to loop by the list of chunks in the document that should be processed.

Network processing is not currently supported by Python scripting, so script will start the process on the single computer only.
Best regards,
Alexey Pasumansky,
Agisoft LLC

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14927
    • View Profile
Re: Python API run batch file (start process on cluster)
« Reply #2 on: March 23, 2016, 06:09:55 PM »
Hello Kisioj,

In PhotoScan Pro v 1.2.4 we have added NetworkClient() and NoetworkTask() classes to the Python API that allow to create batch processing jobs from Python scripts. Also Run Script command has been added to Batch Process dialog.

Here's a brief example of Align Photos job distribution, started from Python script:
Code: [Select]
import PhotoScan

client = PhotoScan.NetworkClient()
doc = PhotoScan.Document()
doc.open("...")

task1 = PhotoScan.NetworkTask()
for c in doc.chunks:
    task1.frames.append((c.key,0))
task1.name = "MatchPhotos"
#params should be specified in case they do differ from default values
task1.params['downscale'] = int(PhotoScan.HighAccuracy)
task1.params['keypoint_limit'] = 40000
task1.params['tiepoint_limit'] = 10000
task1.params['select_pairs'] = int(PhotoScan.ReferencePreselection)
task1.params['network_distribute'] = True #fine level task distribution key


task2 = PhotoScan.NetworkTask()
for c in doc.chunks:
    task2.chunks.append(c.key)
task2.name = "AlignCameras"
task2.params['network_distribute'] = True #fine level task distribution key

path = "processing/py/test.psx" #relative path from root folder
client.connect('117.0.0.1') #server ip
batch_id = client.createBatch(path, [task1, task2])

client.resumeBatch(batch_id)
print("Job started...")
Best regards,
Alexey Pasumansky,
Agisoft LLC

sc

  • Jr. Member
  • **
  • Posts: 59
    • View Profile
Re: Python API run batch file (start process on cluster)
« Reply #3 on: March 23, 2016, 08:28:11 PM »
Awesome!
are there any limitations to this? or is it possible to do all major steps over network via python scripts now?

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14927
    • View Profile
Re: Python API run batch file (start process on cluster)
« Reply #4 on: March 23, 2016, 08:50:52 PM »
Hello sc,

You need to keep in mind that processing functions used in the script (like matchPhotos, buildDenseCloud and etc) wouldn't be automatically distributed across the network, as script used in Barth Process will be performed on the single node only. So for complex task you might need a combination of minor scripts (or script containing different sub-functions ran according to the argument used).

So if you are using the script to create a fully automated processing pipeline using network processing, you need also to create RunScript task and pass another script as a task parameter. It means that you cannot simply post some custom code lines after the assigning and running network batches.
Best regards,
Alexey Pasumansky,
Agisoft LLC

m

  • Newbie
  • *
  • Posts: 11
    • View Profile
Re: Python API run batch file (start process on cluster)
« Reply #5 on: November 23, 2016, 05:46:29 PM »
Hi


We want to run network processing for several chunks, each chunk with a limited number of cameras.

First I have a loop that create chunks, import cameras, import camera calibration file, detect markers, import csv with coordinates for markers.

Then I run network processing for the whole project, using MatchPhotos and AlignCameras as discribed in your example in this post.
But network processing starts when everything else in script is done, even if it is written earlier in the code. Processing then overwrite what is processed earlier.

How can I controll when network processing starts and stops?
I also find API documentation for network processing a bit deficient, will the API be updateed with version 1.3?


m

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14927
    • View Profile
Re: Python API run batch file (start process on cluster)
« Reply #6 on: November 23, 2016, 06:34:05 PM »
Hello m,

When you are using Python script to submit the network processing tasks, but still perform some operations with the same project, it may lead to the project corruption, as the same project is accessed by the local script and by the network processing node.

I think it may be better, if you also create a network task for Run Script command, thus having the complete pipeline submitted to the server as network tasks.

Alternatively, you can create a loop that is checking the status of the submitted batch and re-opens the project after it is finished, but I think that the approach with the complete workflow submitted as a network batch is safer.
Best regards,
Alexey Pasumansky,
Agisoft LLC