Any way to scale a task vertically (clusters, Kubernetes) ?

[Octave A.]

I'm reaching the limits of what can be done on AWS with a single VM.

I'm building high resolution orthomosaics, 3d models and point clouds from very large areas using the 1.8.4 Python API (server license) ran within a Docker container. The base pictures weight about 200GB, all in all the project size + outputs + base pictures can go up to 2TB.

The problem I'm running into is that the processing time is very slow for these workloads, even with the most advanced VMs available on AWS (> 10 hours of calculation). Disk usage can generally also not go over 1.5 TB, so I run into storage issues.

I'm wondering if and how I could scale this workload vertically, using clusters or Kubernetes.

Thanks a lot in advance for your insights!