Forum

Author Topic: Export Only Raster after opening a psx file using doc.open()  (Read 2747 times)

darkl1ght

  • Newbie
  • *
  • Posts: 15
    • View Profile
Export Only Raster after opening a psx file using doc.open()
« on: November 03, 2020, 09:31:42 AM »
Hi,

I am writing scripts to complete the work flow (from building several elements like alignment(step1), matching(step2), building DEM(step3), Dense Cloud(step4), Orthomosaic(step5) and finally Tiled Model(step6), and then exporting the Raster File from saved file). I am using Python API version 1.6.5.

My goal is to implement it in such a way, that if for example it fails in step3, then on the next run of the script, since step1 and step2 is already completed, and I am using the previous chunk in the next run, it should start directly from step3 again. This I am trying to implement(still in progress). Any suggestions or points to be taken care of during this?

Now, the main issue that I am facing is,

When I have completed all the steps,I am trying to export Raster File(earlier Orthomosaic), from a pre-saved .psx extension file. I am running a script just to open the file in Metashape Python API and export the Raster File, without performing any other step of processing(that have been already performed and saved in main_file.psx file).

I have tried exporting in the below testing_argument.py file , but it gives error:

Code: [Select]
#!/usr/bin/python3
import sys
import Metashape
import os
from pathlib import Path

current_path = os.getcwd()
parent_path = str(Path(current_path).parent)
save_path = ''
project_name = ''


def main():

    project_folder_name = 'project2'
    # save path
    save_path = parent_path + '/projects/' \
        + project_folder_name + '/' \
        + project_folder_name
    # + '_' \
    # + start_time

    # creating save directory
    os.makedirs(name=save_path, exist_ok=True)

    global project_name
    project_name = 'main_file.psx'

    # creating document
    doc = Metashape.Document()

    # epsg_code
    epsg_code = 'EPSG::32643'

    doc.open(path=save_path + '/' + project_name)

    # adding chunk
    chunk = Metashape.Chunk
    chunk = doc.chunk

    # adding co-ordinate reference system
    # chunk.crs = Metashape.CoordinateSystem(epsg_code)
    raster_compression = Metashape.ImageCompression()
    raster_compression.tiff_compression = Metashape. \
        ImageCompression.TiffCompressionJPEG
    raster_compression.jpeg_quality = 90
    raster_compression.tiff_big = True

    # projection
    projection = Metashape.OrthoProjection()
    projection.crs = Metashape.CoordinateSystem(epsg_code)

    chunk.exportRaster(
        path=parent_path + '/projects/'
        + project_folder_name + '/' + 'export/' + 'orthophoto.tif',
        format=Metashape.RasterFormatTiles,
        image_format=Metashape.ImageFormat.ImageFormatTIFF,
        projection=projection,
        save_world=True,
        save_alpha=True,
        image_description='Orthophoto',
        image_compression=raster_compression,
        white_background=True,
        save_kml=False,
    )

    doc.save(path=save_path + '/' + project_name, chunks=[chunk])

    return


main()

Error:

Code: [Select]
LoadProject: path = /home/xxxx/projects/project2/project2/main_file.psx
loaded project in 0.000677 sec
Traceback (most recent call last):
  File "testing_arguments.py", line 74, in <module>
    main()
  File "testing_arguments.py", line 55, in main
    chunk.exportRaster(
AttributeError: 'NoneType' object has no attribute 'exportRaster'

What am I doing wrong? Or what I am doing is not possible, since this same thing can be done in UI, wherein we can load the project into the Metashape UI application and then export any data we want from the
chunk>(Right click)> Export xyz option


Please let me know if any more information is needed from my end. Waiting for the response.

Thanks,
Ayush
« Last Edit: November 04, 2020, 08:57:15 AM by darkl1ght »

darkl1ght

  • Newbie
  • *
  • Posts: 15
    • View Profile
Re: Export Only Raster after opening a psx file using doc.open()
« Reply #1 on: November 04, 2020, 07:55:52 PM »
Hi,

The error that I get when I perform directly step5, step4, step3, step2 respectively without performing the prev steps(eg for step5, directly perform step5 ie without performing steps 1,2,3,4) in the current execution are as follows:

Code: [Select]
2020-11-04T22:12:04.586 +0530 INFO  [init_doc.py: 82] ****************************************************************************************************
2020-11-04T22:12:04.586 +0530 INFO  [init_doc.py: 83] logger initiated
2020-11-04T22:12:04.586 +0530 INFO  [init_doc.py:436] Options: {'projectName': 'attur', 'geoProjection': 'EPSG::32643', 'exportRaster': 'true',[b] 'lastStep': '5'[/b]}
2020-11-04T22:12:04.586 +0530 INFO  [init_doc.py:437] Processing started
2020-11-04T22:12:04.587 +0530 INFO  [init_doc.py:462] Opening existing file: /home/mapboxuser/projects/attur/attur/main_file.psx
2020-11-04T22:12:04.787 +0530 INFO  [init_doc.py: 89] Retrieve/Add Chunk
2020-11-04T22:12:04.843 +0530 INFO  [init_doc.py:100] Adding Chunk Completed
2020-11-04T22:12:04.844 +0530 INFO  [init_doc.py:396] Raster Extraction Started
2020-11-04T22:12:05.045 +0530 ERROR [init_doc.py:413] Error while exporting Orthomosaic. Error: Null orthomosaic
2020-11-04T22:12:19.580 +0530 INFO  [init_doc.py: 82] ****************************************************************************************************
2020-11-04T22:12:19.580 +0530 INFO  [init_doc.py: 83] logger initiated
2020-11-04T22:12:19.580 +0530 INFO  [init_doc.py:436] Options: {'projectName': 'attur', 'geoProjection': 'EPSG::32643', [b]'lastStep': '4'[/b], 'exportRaster': 'true'}
2020-11-04T22:12:19.580 +0530 INFO  [init_doc.py:437] Processing started
2020-11-04T22:12:19.581 +0530 INFO  [init_doc.py:462] Opening existing file: /home/mapboxuser/projects/attur/attur/main_file.psx
2020-11-04T22:12:19.781 +0530 INFO  [init_doc.py: 89] Retrieve/Add Chunk
2020-11-04T22:12:19.835 +0530 INFO  [init_doc.py:100] Adding Chunk Completed
2020-11-04T22:12:19.836 +0530 INFO  [init_doc.py:358] step5 started
2020-11-04T22:12:19.836 +0530 INFO  [init_doc.py:360] building orthomosaic started
2020-11-04T22:12:20.037 +0530 ERROR [init_doc.py:368] Error in Build Depth Maps Step Null elevation
Traceback (most recent call last):
  File "init_doc.py", line 364, in begin_image_processing
    projection=projection
Exception: Null elevation
2020-11-04T22:12:24.085 +0530 INFO  [init_doc.py: 82] ****************************************************************************************************
2020-11-04T22:12:24.085 +0530 INFO  [init_doc.py: 83] logger initiated
2020-11-04T22:12:24.086 +0530 INFO  [init_doc.py:436] Options: {'exportRaster': 'true', 'geoProjection': 'EPSG::32643', 'projectName': 'attur', [b]'lastStep': '3'[/b]}
2020-11-04T22:12:24.086 +0530 INFO  [init_doc.py:437] Processing started
2020-11-04T22:12:24.086 +0530 INFO  [init_doc.py:462] Opening existing file: /home/mapboxuser/projects/attur/attur/main_file.psx
2020-11-04T22:12:24.286 +0530 INFO  [init_doc.py: 89] Retrieve/Add Chunk
2020-11-04T22:12:24.340 +0530 INFO  [init_doc.py:100] Adding Chunk Completed
2020-11-04T22:12:24.341 +0530 INFO  [init_doc.py:339] step4 started
2020-11-04T22:12:24.341 +0530 INFO  [init_doc.py:341] building DEM started
2020-11-04T22:12:24.542 +0530 ERROR [init_doc.py:348] Error in Build DEM Step Null dense cloud
Traceback (most recent call last):
  File "init_doc.py", line 344, in begin_image_processing
    projection=projection
Exception: Null dense cloud
2020-11-04T22:12:28.030 +0530 INFO  [init_doc.py: 82] ****************************************************************************************************
2020-11-04T22:12:28.030 +0530 INFO  [init_doc.py: 83] logger initiated
2020-11-04T22:12:28.030 +0530 INFO  [init_doc.py:436] Options: {[b]'lastStep': '2'[/b], 'geoProjection': 'EPSG::32643', 'projectName': 'attur', 'exportRaster': 'true'}
2020-11-04T22:12:28.030 +0530 INFO  [init_doc.py:437] Processing started
2020-11-04T22:12:28.030 +0530 INFO  [init_doc.py:462] Opening existing file: /home/mapboxuser/projects/attur/attur/main_file.psx
2020-11-04T22:12:28.231 +0530 INFO  [init_doc.py: 89] Retrieve/Add Chunk
2020-11-04T22:12:28.285 +0530 INFO  [init_doc.py:100] Adding Chunk Completed
2020-11-04T22:12:28.286 +0530 INFO  [init_doc.py:308] step3 started
2020-11-04T22:12:28.286 +0530 INFO  [init_doc.py:310] building depthmaps started
2020-11-04T22:12:28.487 +0530 ERROR [init_doc.py:318] Error in Build Depth Maps Step Empty region
Traceback (most recent call last):
  File "init_doc.py", line 314, in begin_image_processing
    reuse_depth=reuse_depth
Exception: Empty region


In the above logs, that are logged in a file by python logger, I have provided script parameters like this

Code: [Select]
/home/mapboxuser/Downloads/metashape-pro/metashape.sh -r init_doc.py --projectName attur --geoProjection EPSG::32643 --exportRaster true [b]--lastStep 2[/b]
The --lastStep parameter decides from which step should this process run. the previous steps are skipped. They are skipped because they have already been run and I want to run from next step, from last run step in previous process.

Why am I getting each of these errors? Is there a way to avoid these errors?

Thanks in advance

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14004
    • View Profile
Re: Export Only Raster after opening a psx file using doc.open()
« Reply #2 on: November 17, 2020, 09:06:34 PM »
Hello darkl1ght,

I would suggest to remove "chunk = Metashape.Chunk" assignment from the code sample that you have provided. As it seems to be a possible reason of the problem (such assignment is incorrect). If you have any similar chunk assignment lines in the code, please remove them and check if the processing workflow works, or whether it fails with any different error.

From the script logging it seems that the orthomosaic cannot be exported, as it has not been generated (doesn't exist in the chunk) - Null orthomosaic. Orthomosaic fails, as the selected surface (DEM) doesn't exist in the project - Null elevation. DEM cannot be generated from the dense cloud, since there's no dense cloud - Null dense cloud. Depth maps cannot be generated, as the region is either set incorrectly, or doesn't include any tie points -  Empty region (could also happen, if the matching has been completed and tie points do exist, but the camera alignment has not been run).
Best regards,
Alexey Pasumansky,
Agisoft LLC

darkl1ght

  • Newbie
  • *
  • Posts: 15
    • View Profile
Re: Export Only Raster after opening a psx file using doc.open()
« Reply #3 on: November 21, 2020, 03:01:38 PM »
Hi Alexey,

Thanks for your reply. It is the solution for my issue.

Regards,
darkl1ght

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14004
    • View Profile
Re: Export Only Raster after opening a psx file using doc.open()
« Reply #4 on: November 21, 2020, 04:35:00 PM »
Hello darkl1ght,

If there are still any problems with some parts of the script, please let me know.
Best regards,
Alexey Pasumansky,
Agisoft LLC

darkl1ght

  • Newbie
  • *
  • Posts: 15
    • View Profile
Re: Export Only Raster after opening a psx file using doc.open()
« Reply #5 on: November 27, 2020, 10:10:32 AM »
Hi Alexey,

I was trying to export a DEM file using the script below

Code: [Select]
chunk.exportRaster(
                path=export_path + 'dsm.tif',
                format=Metashape.RasterFormatTiles,
                image_format=Metashape.ImageFormat.ImageFormatTIFF,
                projection=projection,
                save_world=True,
                save_alpha=True,
                image_description='DSM',
                white_background=True,
                save_kml=True,
                source_data=Metashape.DataSource.ElevationData
            )

This gives me a tiff file with 32 bit floating type, and good thing is it gives me elevation data(verified from the QGIS). But It cant be uploaded to MAPBOX since MAPBOX supports a 8-bit GeoTIFF file. Also, why has it been grayscale?(It looks bad)

So I tried to raster transform it using palette raster transform using below script:

Code: [Select]
chunk.exportRaster(
                path=export_path + 'dsm.tif',
                raster_transform=Metashape.RasterTransformType.RasterTransformPalette,  #modification here
                format=Metashape.RasterFormatTiles,
                image_format=Metashape.ImageFormat.ImageFormatTIFF,
                projection=projection,
                save_world=True,
                save_alpha=True,
                image_description='DSM',
                white_background=True,
                save_kml=True,
                source_data=Metashape.DataSource.ElevationData
            )

This gives me a raster file with 8-bit unsigned integer data type with a color palette exactly similar to Agisoft GUI, but it does not contain any elevation data(verified from the QGIS).

Is there a way I can get a 8-bit GeoTIFF file with elevation data and the default color palette of Agisoft GUI, not grayscale?

Thanks,
darkl1ght
« Last Edit: November 27, 2020, 10:13:06 AM by darkl1ght »

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14004
    • View Profile
Re: Export Only Raster after opening a psx file using doc.open()
« Reply #6 on: November 27, 2020, 02:33:46 PM »
Hello darkl1ght,

32-bit TIFF with the elevation data is a single-channel raster where each pixel contains a floating point corresponding to the elevation. It is not a grayscale image, the visual color representation should be adjusted in the application that you are using to view the file.

Best regards,
Alexey Pasumansky,
Agisoft LLC

darkl1ght

  • Newbie
  • *
  • Posts: 15
    • View Profile
Re: Export Only Raster after opening a psx file using doc.open()
« Reply #7 on: November 27, 2020, 07:53:22 PM »
HI Alexey,

This does not answer my main question. Can I get an exported TIFF file that is 8-bit and a different color palette along with elevation data?

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14004
    • View Profile
Re: Export Only Raster after opening a psx file using doc.open()
« Reply #8 on: November 28, 2020, 04:56:30 PM »
Hello darkl1ght,

Do you mean that you need to export DEM as four channel TIFF file with RGB channels and elevation data in the fourth channel?

Do you have any example of data in such representation? Unless I am mistaken, TIFF format doesn't allow to have different data types for the channels in the same file, therefore all four bands in such case should be either 8-bit or 32-bit. And I have some doubts that 8-bit float would allow to save the elevation values with proper grade of accuracy.
Best regards,
Alexey Pasumansky,
Agisoft LLC

darkl1ght

  • Newbie
  • *
  • Posts: 15
    • View Profile
Re: Export Only Raster after opening a psx file using doc.open()
« Reply #9 on: November 28, 2020, 07:00:06 PM »
Hi Alexey,
Firstly, Thanks for the quick replies.

When I open the psx file in agisoft UI, it shows something like this

(image1.png) https://drive.google.com/file/d/1osAxn-shuCa9WuJnAJXnLT5fB-jLmaX-/view

I just know that it represents a RGB image with Elevation data(ignore the contours).

Now when I simply export this using the below scripts, it exports and looks like this((image2.png) https://drive.google.com/file/d/128SxTGwC1qVOuAyHY_4X2oL4AO4ASuA2/view?usp=sharing) in QGIS.

Script:
Code: [Select]
chunk.exportRaster(
                path=export_path + 'dsm.tif',
                format=Metashape.RasterFormatTiles,
                image_format=Metashape.ImageFormat.ImageFormatTIFF,
                projection=projection,
                save_world=True,
                save_alpha=True,
                image_description='DSM',
                white_background=True,
                save_kml=True,
                source_data=Metashape.DataSource.ElevationData
            )

As per your reply,

Hello darkl1ght,

32-bit TIFF with the elevation data is a single-channel raster where each pixel contains a floating point corresponding to the elevation. It is not a grayscale image, the visual color representation should be adjusted in the application that you are using to view the file.



Then why it is not RGB and is in two colors as can be seen from QGIS screenshot(image2.png)

Now, when I export it using raster transform, by applying palette transform,

I get below image which is RGB, but does not contain elevation information as evident from screenshot (image3.png)https://drive.google.com/file/d/1qtVeAnQ8NOLbamBHJqqykG2lDsAbOmQN/view?usp=sharing

For images 2 and 3, I have just opened them in QGIS, no modifications made whatsoever.

Image 2 is 32-bit Float.

Image 3 is 8-bit RGB image


Hello darkl1ght,

Do you mean that you need to export DEM as four channel TIFF file with RGB channels and elevation data in the fourth channel?
-->  I dont know if that is possible or not, but it can be seen in Image1, Agisoft UI has RGB + elevation data.

Do you have any example of data in such representation? Unless I am mistaken, TIFF format doesn't allow to have different data types for the channels in the same file, therefore all four bands in such case should be either 8-bit or 32-bit. And I have some doubts that 8-bit float would allow to save the elevation values with proper grade of accuracy.
-->I am not that expert in image processing, so I don't know if I can answer that. Its okay if TIFF does not allow that to happen, but is there any other way to export the DEM that allows it?



Also, if nothing works out here, I need an output from Agisoft that can give me elevation data of DEM model, that can be successfully uploaded to MAPBOX to access elevation data. Would appreciate any help/guidance in that direction as well.

Thanks,
darkl1ght
« Last Edit: November 28, 2020, 07:03:54 PM by darkl1ght »

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14004
    • View Profile
Re: Export Only Raster after opening a psx file using doc.open()
« Reply #10 on: November 28, 2020, 09:34:43 PM »
Hello darkl1ght,

The elevation model internally contains the altitude information only, in the Ortho view mode Metashape applies the pseudo colors with certain RGB palette (where pure blue represents the min altitude and pure red - max altitude). The data is not stored as RGB + altitude.

The visualization of elevation model data in GIS application can be customized with the variety of instruments. Please check the following article, how to assign pseudo colors to the elevation model (single band floating point values) in QGIS:
https://docs.qgis.org/3.4/en/docs/training_manual/rasters/changing_symbology.html
The default representation is "singleband gray" - min altitude is pure black (zeros for RGB pseudocolors) and max altitude - pure white (max values for RGB values). For the intermediate values RGB pseudo color values are identical (I mean their value for certain altitude), which means that the resulting image is shown in the grayscale.

To keep the absolute altitude information during the export you should export DEM in default mode: as floating point value TIFF without applying raster transformation.
Best regards,
Alexey Pasumansky,
Agisoft LLC