Forum

Author Topic: Dense cloud not in project after processing  (Read 4122 times)

mks_gis

  • Newbie
  • *
  • Posts: 40
    • View Profile
Dense cloud not in project after processing
« on: February 14, 2018, 01:28:28 AM »
Hi,

 I'm processing images from UAV to generate DEM and Orthos. I wrote a Python script to process multiple chunks at the same time (and to learn using Python for PS).

 I'm now using the script and in the build dense cloud step after processing for several hours the dense cloud is not there, even though the log file says it ran successfully. Depth maps don't seem to be there either.  I'm on 1.4.

This has worked in a previous project with multiple chunks, although one one occasion I had the same problem, but on a smaller image set and I just ran it again, this time successfully. I run each step (align, dense, dem, ortho, export) separately, as I check the results and clean up before continuing.

Why is the Dense Cloud missing, what did I miss?

Cheers
Martin


code snippet that runs dense cloud:
def dense_c(self):
        for _ in self.cks:
            _.buildDepthMaps(quality=PhotoScan.HighQuality,
                                filter=PhotoScan.MildFiltering)           
            _.buildDenseCloud(point_colors=True)


Log (last few lines):
2018-02-13 16:45:27
2018-02-13 16:45:27 Depth reconstruction devices performance:
2018-02-13 16:45:27  - 5%    done by CPU
2018-02-13 16:45:27  - 95%    done by GeForce GTX 980
2018-02-13 16:45:27 Total time: 6356.12 seconds
2018-02-13 16:45:27
2018-02-13 16:45:28 Finished processing in 6375.36 sec (exit code 1)
2018-02-13 16:45:28 BuildDenseCloud
2018-02-13 16:45:28 Generating dense cloud...
2018-02-13 16:45:28 Generating dense point cloud...
2018-02-13 16:45:32 selected 284 cameras in 4.853 sec
2018-02-13 16:45:32 working volume: 8580x3829x2775
2018-02-13 16:45:32 tiles: 2x1x1
2018-02-13 16:45:32 selected 135 cameras in 0.001 sec
2018-02-13 16:45:32 preloading data... done in 3.479 sec
2018-02-13 16:45:36 filtering depth maps... done in 881.324 sec
2018-02-13 17:00:18 preloading data... done in 7.266 sec
2018-02-13 17:00:25 accumulating data... done in 12.639 sec
2018-02-13 17:00:39 building point cloud... done in 2.298 sec
2018-02-13 17:00:42 selected 232 cameras in 0.001 sec
2018-02-13 17:00:42 preloading data... done in 5.479 sec
2018-02-13 17:00:47 filtering depth maps... done in 10398 sec
2018-02-13 19:54:06 preloading data... done in 13.742 sec
2018-02-13 19:54:20 accumulating data... done in 29.615 sec
2018-02-13 19:54:51 building point cloud... done in 2.034 sec
2018-02-13 19:54:54 52859925 points extracted
2018-02-13 19:54:55 Finished processing in 11367.2 sec (exit code 1)

Full code:
import PhotoScan
import os

# Checking compatibility
compatible_major_version = "1.4"
found_major_version = ".".join(PhotoScan.app.version.split('.')[:2])
if found_major_version != compatible_major_version:
    raise Exception("Incompatible PhotoScan version: {} != {}".format(found_major_version, compatible_major_version))

class ChunksProc(object):
   
    def __init__(self, cks):
        self.cks = cks

    def remove_align(self):
        for _ in self.cks:
            for c in _.cameras:
                c.transform = None
               
    def align(self):
        for _ in self.cks:
            _.matchPhotos(accuracy=PhotoScan.HighAccuracy,
                            generic_preselection=False,
                            reference_preselection=True,
                            filter_mask=True,
                            keypoint_limit=40000,
                            tiepoint_limit=0)
            _.alignCameras()

    def dense_c(self):
        for _ in self.cks:
            _.buildDepthMaps(quality=PhotoScan.HighQuality,
                                filter=PhotoScan.MildFiltering)           
            _.buildDenseCloud(point_colors=True)

    def dem(self):
        for _ in self.cks:
            _.buildDem(source=PhotoScan.DenseCloudData,
                        interpolation=PhotoScan.EnabledInterpolation)

    def ortho(self):
        for _ in self.cks:
            _.buildOrthomosaic(surface=PhotoScan.ElevationData,
                                blending=PhotoScan.MosaicBlending,
                                fill_holes=False)

    def export(self, pth, prefix):
        for _ in self.cks:
            num = str(_)[-3]
            file = '{}_DSM_{}.tif'.format(prefix, num)
            _.exportDem(path = os.path.join(pth, file),
                        write_world = True)
            file = '{}_Ortho_{}.tif'.format(prefix, num)
            _.exportOrthomosaic(path = os.path.join(pth, file),
                                write_world = True)

    def exp_jpg(self, pth, prefix, w = None, h = None):
        doc.save()
        for _ in self.cks:
            num = str(_)[-3]
            file = '{}_Ortho_{}.jpg'.format(prefix, num)
            _.exportOrthomosaic(path = os.path.join(pth, file),
                                write_world = True)   

doc = PhotoScan.app.document
if not len(doc.chunks):
        raise Exception("No chunks!")
cks = ChunksProc(doc.chunks)
pth = r'E:\Dominica\Data Output Folder'
jpgpth = r'E:\Dominica\Data Output Folder\JPG Orthos'
prefix = input('Enter file prefix: ')
cks.align()
doc.save()
cks.dense_c()
doc.save()
cks.dem()
doc.save()
cks.ortho()
doc.save()
cks.export(pth, prefix)
cks.exp_jpg(jpgpth, prefix)
doc.save()

mks_gis

  • Newbie
  • *
  • Posts: 40
    • View Profile
Re: Dense cloud not in project after processing
« Reply #1 on: February 14, 2018, 12:20:21 PM »
Hi,

I've rewritten my code and it now works, although I don't know what I did to fix it. It seems to have been related to only having one chunk in this particular project:

import PhotoScan
import os

# Checking compatibility
compatible_major_version = "1.4"
found_major_version = ".".join(PhotoScan.app.version.split('.')[:2])
if found_major_version != compatible_major_version:
    raise Exception("Incompatible PhotoScan version: {} != {}".format(found_major_version, compatible_major_version))

class PS_Proc(object):
   
    def __init__(self, doc):
        self.doc = doc
        if not len(self.doc.chunks):
                raise Exception("No chunks!")
        self.chunks = self.doc.chunks

    def remove_align(self):
        for _ in self.chunks:
            for c in _.cameras:
                c.transform = None
               
    def align(self):
        for _ in self.chunks:
            _.matchPhotos(accuracy=PhotoScan.HighAccuracy,
                            generic_preselection=False,
                            reference_preselection=True,
                            filter_mask=True,
                            keypoint_limit=40000,
                            tiepoint_limit=0)
            _.alignCameras()
            self.doc.save()

    def dense_c(self):
        for _ in self.chunks:
            _.buildDepthMaps(quality=PhotoScan.HighQuality,
                                filter=PhotoScan.MildFiltering)           
            _.buildDenseCloud(point_colors=True)
            self.doc.save()
           
    def dem(self):
        for _ in self.chunks:
            _.buildDem(source=PhotoScan.DenseCloudData,
                        interpolation=PhotoScan.EnabledInterpolation)
        self.doc.save()               

    def ortho(self):
        for _ in self.chunks:
            _.buildOrthomosaic(surface=PhotoScan.ElevationData,
                                blending=PhotoScan.MosaicBlending,
                                fill_holes=False)
        self.doc.save()
                               
    def export(self, pth, prefix):
        for _ in self.chunks:
            num = str(_)[-3]
            file = '{}_DSM_{}.tif'.format(prefix, num)
            _.exportDem(path = os.path.join(pth, file),
                        write_world = True)
            file = '{}_Ortho_{}.tif'.format(prefix, num)
            _.exportOrthomosaic(path = os.path.join(pth, file),
                                write_world = True)

    def exp_jpg(self, pth, prefix, w = None, h = None):
        doc.save()
        for _ in self.chunks:
            num = str(_)[-3]
            file = '{}_Ortho_{}.jpg'.format(prefix, num)
            _.exportOrthomosaic(path = os.path.join(pth, file),
                                write_world = True)   

ps_doc = PS_Proc(PhotoScan.app.document)
pth = r'E:\Dominica\Data Output Folder'
jpgpth = r'E:\Dominica\Data Output Folder\JPG Orthos'
prefix = input('Enter file prefix: ')
ps_doc.align()
ps_doc.dense_c()
ps_doc.dem()
ps_doc.ortho()
ps_doc.export(pth, prefix)
ps_doc.exp_jpg(jpgpth, prefix)
doc.save()

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14813
    • View Profile
Re: Dense cloud not in project after processing
« Reply #2 on: February 14, 2018, 05:18:10 PM »
Hello Martin,

I'll try to reproduce the issue using your code, but can you please confirm that the provided log is related to the chunk with the missing dense cloud, whereas the alignment has been properly completed for it?
Best regards,
Alexey Pasumansky,
Agisoft LLC

mks_gis

  • Newbie
  • *
  • Posts: 40
    • View Profile
Re: Dense cloud not in project after processing
« Reply #3 on: February 15, 2018, 12:28:54 PM »
Hi Alexey,

 Thanks for getting back to me. The alignment completed properly and the log file is for the chunk with the missing dense cloud.

 I've tried to reproduce the issue with a subset of images, with the same code and I can't reproduce it now. It was most likely related to having only a single chunk; I had written this code to work with multiple chunks originally. I've since rewritten the code, making the doc the basis for the class and it's working fine. It was a glitch at the time, but I'm not sure it's reproducible. I would not want to waste your time for something I most likely did wrong. But I'm happy to learn if you can point out my errors.

Cheers
Martin


import PhotoScan
import os

# Checking compatibility
compatible_major_version = "1.4"
found_major_version = ".".join(PhotoScan.app.version.split('.')[:2])
if found_major_version != compatible_major_version:
    raise Exception("Incompatible PhotoScan version: {} != {}".format(found_major_version, compatible_major_version))

class PS_Proc(object):
   
    def __init__(self, doc):
        self.path = PhotoScan.app.getExistingDirectory('Specify TIF DSM/Ortho export folder:')
        self.prefix = PhotoScan.app.getString(label='Enter file prefix: ', value='')
        self.doc = doc
        if not len(self.doc.chunks):
                raise Exception("No chunks!")
        self.chunks = self.doc.chunks

    def remove_align(self):
        for _ in self.chunks:
            for c in _.cameras:
                c.transform = None
               
    def align(self):
        for _ in self.chunks:
            _.matchPhotos(accuracy=PhotoScan.HighAccuracy,
                            generic_preselection=False,
                            reference_preselection=True,
                            filter_mask=True,
                            keypoint_limit=40000,
                            tiepoint_limit=0)
            _.alignCameras()
            self.doc.save()

    def dense_c(self):
        for _ in self.chunks:
            _.buildDepthMaps(quality=PhotoScan.HighQuality,
                                filter=PhotoScan.MildFiltering)           
            _.buildDenseCloud(point_colors=True)
            self.doc.save()
           
    def dem(self):
        for _ in self.chunks:
            _.buildDem(source=PhotoScan.DenseCloudData,
                        interpolation=PhotoScan.EnabledInterpolation)
        self.doc.save()               

    def ortho(self):
        for _ in self.chunks:
            _.buildOrthomosaic(surface=PhotoScan.ElevationData,
                                blending=PhotoScan.MosaicBlending,
                                fill_holes=False)
        self.doc.save()
                               
    def export(self):
        for _ in self.chunks:
            num = str(_)[-3]
            file = '{}_DSM_{}.tif'.format(self.prefix, num)
            _.exportDem(path = os.path.join(self.path, file),
                        write_world = True)
            file = '{}_Ortho_{}.tif'.format(self.prefix, num)
            _.exportOrthomosaic(path = os.path.join(self.path, file),
                                write_world = True)

    def exp_jpg(self):
        jpgpath = os.path.join(self.path, 'JPG Orthos')
        if not os.path.exists(jpgpath):
            jpgpath = PhotoScan.app.getExistingDirectory('Specify JPG export folder:')
        for _ in self.chunks:
            num = str(_)[-3]
            file = '{}_Ortho_{}.jpg'.format(self.prefix, num)
            _.exportOrthomosaic(path = os.path.join(jpgpth, file),
                                write_world = True)   
    def run_all(self):
        self.align()
        self.dense_c()
        self.dem()
        self.ortho()

    def run_post_align(self):
        self.dense_c()
        self.dem()
        self.ortho()   

def menu(label, function):
    PhotoScan.app.addMenuItem(label, function)
    print('To execute this script press {}'.format(label))

#initiate object   
ps_doc = PS_Proc(PhotoScan.app.document)
#create menu options
menu('Custom/Remove alignment(optional)', ps_doc.remove_align)
menu('Custom/Align', ps_doc.align)
menu('Custom/Build dense cloud', ps_doc.dense_c)
menu('Custom/Build DSM', ps_doc.dem)
menu('Custom/Build Ortho', ps_doc.ortho)
menu('Custom/Run all', ps_doc.run_all)
menu('Custom/Run all after alignment', ps_doc.run_post_align)
menu('Custom/Export TIF', ps_doc.export)
menu('Custom/Export JPG', ps_doc.exp_jpg)


           

luminus

  • Newbie
  • *
  • Posts: 13
    • View Profile
Re: Dense cloud not in project after processing
« Reply #4 on: June 28, 2018, 04:32:03 AM »
I'm seeing a similar behavior on versions 1.3 and 1.4. I have a script that does the alignment step (matchPhotos then alignCameras) and it clearly spends a lot of time processing and the logs show it works, but when it finishes, there are no tie points and the cameras don't look aligned.

If you do the same steps over again but with the UI directly, it works.

This has been observed in both 1.3 and 1.4, and I have a setup where I switch between them occasionally (primarily for testing and development).

luminus

  • Newbie
  • *
  • Posts: 13
    • View Profile
Re: Dense cloud not in project after processing
« Reply #5 on: June 28, 2018, 09:57:13 PM »
The issue in my case was the frequent use of save( path ) for an already open project. Thanks to Agisoft support, it was identified this was causing the chunk objects to be invalidated. Only save() should be used if there is an open project and the path isn't being changed.

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14813
    • View Profile
Re: Dense cloud not in project after processing
« Reply #6 on: July 01, 2018, 06:14:03 PM »
Hello luminus,

The doc.save(save_path) operation should work similarly to doc.save() operation if the document is already saved and doc.path is identical to save_path.

But it is still suggested to save the document for the first time before making assignment to the bottom level variables (like chunks in the document, for example).

If anyone is facing similar problems in future, then the suggested sequence of commands is:
Code: [Select]
doc = PhotoScan.Document() #or doc = Photoscan.app.document
doc.save(path)
chunk =  doc.addChunk()
#etc.
Instead of saving the data in "untitled" document after certain operations already applied to the data. Alternatively, it will be required to re-assign chunks-related variables are first document save.
Best regards,
Alexey Pasumansky,
Agisoft LLC