Anybody tried to run Metashape with python Celery?

[koriel-angelswing]

Hi,

I am running Metashape with celery to run Metashape's build jobs one by one serially.
Celery is job queue and worker which is integrated with Redis. Celery runs worker which process actual job and it forks child process using (fork() function) in which some procedures run (functions or something).

However for some reasons, Metashape's buildDepthMaps job freezes in that child process with no errors. On the other hand, matchPhotos and alignCameras run very well.

I want to hear from people who tried like this before and get some insights.

[koriel-angelswing]

Hi,

I am running Metashape with celery to run Metashape's build jobs one by one serially.
Celery is job queue and worker which is integrated with Redis. Celery runs worker which process actual job and it forks child process using (fork() function) in which some procedures run (functions or something).

However for some reasons, Metashape's buildDepthMaps job freezes in that child process with no errors. On the other hand, matchPhotos and alignCameras run very well.

I want to hear from people who tried like this before and get some insights.

Replying to myself.

If somebody wants to use Metashape with Celery, use gevent when running worker.

[adamjg]

Hi Koriel,

I would be interested to know more about your current setup as I have been trying to achieve something similar recently. IE a flask app which queues tasks and celery running in the background.

The thing im mostly confused about is how the Metashape library deals with the threads as even with just 1 worker it utilises all available cores?

[koriel-angelswing]

Hi Koriel,

I would be interested to know more about your current setup as I have been trying to achieve something similar recently. IE a flask app which queues tasks and celery running in the background.

The thing im mostly confused about is how the Metashape library deals with the threads as even with just 1 worker it utilises all available cores?

I made Metashape Flask server but what if 100 requests come at the same time? A hundred of depth map, dem or others build will be triggered. This is not efficient. So I separated Metashape's jobs into heavy, middle and light.

Build job belongs to heavy queue.
Export job belongs to middle queue.
Marking and others which occupy a little computing source belong to light queue.

Here, I launch 4 processes (flask app, heavy worker, middle worker, light worker) and I set the number of jobs each worker can process at once.

Heavy: 1
Middle: 20
Light: 100

By this architecture, this is scalable, if you want to assign more computing resources to Metashape Flask app, you can just add one more computer and run three workers which fetches the job from queue(redis) and process.