long path bug in Unix Network exportRaster: non-network & Win10 net/non-net work

[andyroo]

[EDIT 2 - this was my screw-up, not a bug, and I corrected the code - TLDR; the code below works now, original code (now deleted) had document path, not the PSX itself, in the batch_id ]

I wrote a script to loop through chunks in a psx, and for each chunk with a default (checked) DEM, it will get the extent and export DEMs with the bounding box/BBox rounded to some multiple of the specified DEM export resolution.

I designed the script to work in either network or non-network mode, and tested it in both modes on a Win10 machine (tested network with node/monitor/GUI/host all on 127.0.0.1). It looks for app.settings.network_enable = True and runs in network mode if True, standalone if not. On the Windows machine I was able to generate DEMs from multiple chunks as expected.

When I tried it in network mode on our unix machines, I got Error: Can't read file: Is a directory (21): and I have no idea why.

In non-network mode it runs just fine. It kind of seems like the network task is truncating the file length or something, but the network task looks fine to me. The total path length including filename was 154 characters and I've attached a screenshot showing the bad script run on a node plus several attempts to duplicate the filename at the end, the last one was successful and the extra comma is because I pasted it into the filename apparently (created a file with a comma in the extension, which I didn't even know was legal).

[edit accidentally hit post before attaching image]

Code: [Select]

'''
make bounding boxes and build integer bounded DEMs for ALL default DEMs in the open PSX file
aritchie@usgs.gov 2021-05-03 tested on Metashape 1.7.1

This script creates a bounding box from the extent of the existing default full resDEM, rounded to the specified interval, then creates a raster with a specified resolution,
FOR EVERY DEFAULT DEM IN EVERY CHUNK IN THE PSX.

Raster will be placed in a user-specified (via script variable) subdirectory of the existing project ('dem' by default)
DIRECTORY WILL BE CREATED IF IT DOESN'T EXIST.
A user-specified suffix will be appended to the chunk label (in user variables below)
---CAUTION THERE IS NO ERROR CHECKING FOR LEGAL FILENAMES----
There is no error checking in the script. It will throw errors if there is no default DEM,

If there are bad filename characters, etc. I have NO idea what will happen. Be careful.

Andy
'''
import Metashape
import math
import os
from os import path
#-------------------------------------------------------#
#define user-set variables
raster_rounding_multiple = 10   # Default = 10 - This will be the multiple that the raster resolution is multiplied by to define the units the min/max extents are rounded to
raster_resolution = 1           # Default = 1 - cell size of exported DEM
raster_crop = True              # Default = True - True means Bounding Box in rounded IN - minimum extent is rounded up and maximum extent is rounded down from raster edges. False is reversed
                                # TODO - make it so metashape checks to see if this is an interpolated raster (shrink) or uninterpolated (grow?)
                                # ALSO - maybe we want to project the xy coordinates of the 3D dense cloud region and use those instead? this will result in no/minimal collar though...
dem_subdir = 'dem_20210504'              # this is a subdir that will be created under the document (PSX) path
dem_suffix = '_NAD83_2011_NAVD88_UTM18'

#-----OPERATIONAL CODE IS BELOW. EDIT AT YOUR PERIL-----#
raster_rounding_interval = raster_rounding_multiple * raster_resolution
app = Metashape.app
doc = app.document
network_tasks = list()
for chunk in doc.chunks:
    if chunk.elevation:
        print(chunk.label)
        out_projection = chunk.elevation.projection
        compression = Metashape.ImageCompression()
        compression.tiff_compression = Metashape.ImageCompression.TiffCompressionLZW
        compression.tiff_big = True
        compression.tiff_overviews = True
        compression.tiff_tiled = True
           
        def round_down(x):
            return int(raster_rounding_interval * math.floor(float(x)/raster_rounding_interval))

        def round_up(x):
            return int(raster_rounding_interval * math.ceil(float(x)/raster_rounding_interval))


        testbox = Metashape.BBox() #create a bounding box for the raster
        print('')
        print('original DEM BBox coordinates:')
        print('min: ', Metashape.Vector((min(chunk.elevation.left, chunk.elevation.right), min(chunk.elevation.bottom, chunk.elevation.top))))
        print('max: ', Metashape.Vector((max(chunk.elevation.left, chunk.elevation.right), max(chunk.elevation.bottom, chunk.elevation.top))))

        if raster_crop:
            testbox.min = Metashape.Vector((round_up(min(chunk.elevation.left, chunk.elevation.right)), round_up(min(chunk.elevation.bottom, chunk.elevation.top))))
            testbox.max = Metashape.Vector((round_down(max(chunk.elevation.left, chunk.elevation.right)), round_down(max(chunk.elevation.bottom, chunk.elevation.top))))
        else:
            testbox.min = Metashape.Vector((round_down(min(chunk.elevation.left, chunk.elevation.right)), round_down(min(chunk.elevation.bottom, chunk.elevation.top))))
            testbox.max = Metashape.Vector((round_up(max(chunk.elevation.left, chunk.elevation.right)), round_up(max(chunk.elevation.bottom, chunk.elevation.top))))

        if raster_crop:
            print('extent was SHRUNK to: ')
            print('min: ',testbox.min)
            print('max: ',testbox.max)
        else:
            print('extent was GROWN to: ')
            print('min: ',testbox.min)
            print('max: ',testbox.max)

        doc_path = os.path.split(doc.path)[0]
        outPath = os.path.normpath(doc_path + os.sep + dem_subdir)

        outFilename = chunk.label + dem_suffix + '_' + str(raster_resolution) + 'm' + '.tif'
        exportFile = os.path.normpath(outPath+os.sep+outFilename)
        if not os.path.exists(outPath):
            print('testing create path: ' + outPath)
            os.makedirs(outPath)
            print('testing file writestring: ' + exportFile)
        else:
            if not os.path.isfile(exportFile):
                print('testing file writestring: ' + exportFile)
        #
        if not app.settings.network_enable:
            chunk.exportRaster(path = exportFile, image_format=Metashape.ImageFormatTIFF, projection = out_projection, region = testbox, resolution_x = raster_resolution,  resolution_y = raster_resolution, image_compression=compression, save_world = False, white_background = False,source_data = Metashape.ElevationData)
        else:
            task = Metashape.Tasks.ExportRaster()
            task.path = str(exportFile)
            task.image_compression = compression
            task.image_format = Metashape.ImageFormatTIFF
            task.projection = out_projection
            task.region = testbox
            task.resolution_x = raster_resolution
            task.resolution_y = raster_resolution
            task.save_world = False
            task.source_data = Metashape.ElevationData

            n_task = Metashape.NetworkTask()
            n_task.name = task.name
            n_task.params = task.encode()
            n_task.frames.append((chunk.key, 0))
            network_tasks.append(n_task)
    else:
        print(chunk.label, ' has no DEM.')

if app.settings.network_enable:
    client = Metashape.NetworkClient()
    client.connect(app.settings.network_host) #server ip
    batch_id = client.createBatch(doc.path, network_tasks)
    client.resumeBatch(batch_id)
print('script complete')



[Alexey Pasumansky]

Hello Andy,

Were there actual difference between the unsuccessful and successful export attempts displayed in the log?

Is it possible, that there are some unexpected symbols in the exportFile path string, like end of line or anything similar? Maybe something wrong with os.sep on your Linux system?

[andyroo]

agh. My bad. Apparently the problem was this line:

 batch_id = client.createBatch(doc_path, network_tasks)

because doc_path was the path to the .PSX without the filename of the project. I'm guessing it worked on my Windoze system because the project was still open in the GUI on the same machine that was the node, or something like that. I fixed the issue by changing it to doc.path.