Automated PS script to process single or multiple chunks - opinions please

[mks_gis]

***Please see later posts for updated script

Hi,

 I started a script a while ago to automate some steps, which turned into a full blown automated script, including gradual selection. I use it for UAV and close range work.

 I'd like some opinions from you good people as to the underlying processing logic. I often have to run multiple models with different settings, so I create one PS doc and have multiple chunks, although it does work with just one chunk. I set up the images, coord sys for each chunk, add GCP where applicable and then, so I don't need to supervise, I just run the script, leave it for the weekend and have all data ready come Monday. Seems to work...

The script creates menu options to cater to my specific needs, but can be adapted.

Some points I could use some clarification on:
I use adaptive, although it is now set to False by default (http://www.agisoft.com/forum/index.php?topic=9401.0). I'm not sure why that was changed and when it should be used.
 
I use optimizeCameras without parameters, trusting PS to select the right ones. Should I set parameters?

My automated gradual selection process is:
Reconstruction uncertainty to 10 - optimize
Projection Accuracy to 10 (models) or 3 (UAV) - optimize
Reprojection error - self-adjusting to find the right value to leave me with 10% of the original points - optimize

Suggestions very welcome
Cheers
M

[mks_gis]

***Please see later posts for updated script

I've tweaked the optional grad sel process:
1. rec uncert of 10, unless > 50% are selected, then self adjusting - optimise
2. proj acc self adjusting to select < 50% of ties left after 1 - optimise
optional break - allow for adding GCP
3. repro error self adjusting in 10% steps/optimise to select up to 80% of the initial ties.

Loosely based on USGS workflow discussed here: http://www.agisoft.com/forum/index.php?topic=9485.msg44134#msg44134

Seems to work with P4 images, Sony a6000 UVA images and DSLR close range models to reduce pixel errors. I know it's not a substitute for trying to tweak individual models for max accuracy, but good for quickly processing to a good standard, I believe. Happy to take comments.

Also added logging of values used in each step for reference if needed later.

[mks_gis]

Spotted a mistake, gradual selection currently only works for single chunks...

[mks_gis]

fixed

needs to run as one session though, i.e if you've run the script keep the session open for grad sel process

[Hydro_Ydé]

Thank you Mks_gis for sharing this ! Have to try this.

Quick stop after "log file to check progress :" followed by a mixed / & \ file name.

[ahyakbaba]

thanks for your really useful posts. also may i have your workflows as you put your custom menu for each command

[mks_gis]

Currently needs to be run as one session, valid ties not accessible after
    gradual selection. To be fixed in next version using tracks.
   
    len(chunk.point_cloud.points) - valid ties
    len(chunk.point_cloud.tracks) - all ties

With gradual selection:
    After alignment the model is optimised and then a gradual selection is run.
    After the align process all tie points have errors attached
            to them. The values used are a matter of debate.
            Rec Uncert is normally set to 10 and run only once. If
            the value of 10 selects more than 50% of the tie points the value
            self-adjusts to less than 50%.
            Proj acc is set to start at 2 and self adjusts so less
            than 50% of the tie points left after Rec Uncert are selected.
            For Reprojection Error the aim is to select up to a set percentage
            (default 80) of the initial tie points. The value used is self
            selecting. The selection process is in steps of 10% and optimisation
            is run after each step.
            Values used are saved in the log file.
           
Without gradual selection:
    After alignment the model is optimised once.

3D Model/Geo: Geo basically is anything where the photos have coordinates or you are using Ground Control Points(GCP). If you have photos of objects without that info use 3D Model

Set-up workflow:
1. Set up your chunk(s)
    import photos
    name chunks (optional - helps in sorting the output)
    save a copy of the PS Doc

Custom workflow:
2. (Geo) Set reference system for each chunk
3. Load script
    chose output folder
    chose file prefix (this gets added to each output file - helps if trying different settings)
4. Under "Change Values" select "Get Current Parameter info" (check console window for output)
    check image quality threshold
    check filtering (options: Mild (default), Moderate, Aggressive, none)
    (Geo) check export CRS (in case you want the output to be different from the CRS of the camera/GCP)

No GCP or checking alignment - runs whole process and exports:
5a. (Geo) Choose "Custom":"Run - Geo" (with or without gradual selection)
or
5b. (3D Model) Choose "Custom":"Run - 3D Model" (with or without gradual selection)

With GCP or scales or if wanting to check alignment:
5. Choose "Custom":"Align only" (with or without gradual selection)
6. Set up GCP for each chunk or set scale
7. Check alignment
8a. (Geo) Choose "Custom":"Run all after alignment - Geo" (with or without gradual selection)
or
8b. (3D Model) Choose "Custom":"Run all after alignment - 3D Model" (with or without gradual selection)

Files generated depend on method used (3D or Geo) and input photos.

You can run methods individually as well after opening the script, in console type "ps_doc." and then hit "TAB" to see the options.

[LFSantosgeo]

Hey mks_gis nice coding.

    len(chunk.point_cloud.tracks) - counts valid and invalid tiepoints right?
    len(chunk.point_cloud.points) - and this counts the initial number of valid tiepoints before gradual selection

Have you managed to count tiepoints with Python after gradual selection? Every time I use len(chunk.point_cloud.points) I get the same initial value

Currently needs to be run as one session, valid ties not accessible after
    gradual selection. To be fixed in next version using tracks.
   
    len(chunk.point_cloud.points) - valid ties
    len(chunk.point_cloud.tracks) - all ties

[LFSantosgeo]

Just found out. Python tricks...

Code: [Select]

points = point_cloud.points
len(points)Gives the total valid points.
Then Gradual selection is done.
And then:

Code: [Select]

chunk = doc.chunk
len(point_cloud.points)Gives the valid tiepoints number after gradual selection and for that you can assign to a new variable. You could reassign the variable points but you will loose the total points value before gradual selection.

[mks_gis]

Hi Luiz,

 thanks for commenting. PhotoScan doesn't seem to update the points until the model is optimised.

 The approach in your second post works, but only until the session is shut down. Tracks would allow you to re-open the document later and carry on. But tracks give all points, what I'm really after is the valid points after alignment. I might not change to tracks because of this. I've made a few changes to the code and am testing, will possibly upload a newer version soon-ish.

Cheers

[LFSantosgeo]

Hey mks_gis,

What do you think of the following code structure, for instance, to range from a value to another of filter levels and try to remove points if a condition is True? It also can restart searching through the same range values if one is found.  I'm working on still, just thought of sharing this idea to implement filtering for the USGS workflow (thinking of reconstruction uncertainty running more than once till no points are selected based on the chosen range). This of course would rapidly decrease points in point cloud.

Code: [Select]

threshold = range(0, 11, 1)  # here would be the filter levels range
listx = []  # throwing a list just for the exemple
for i in threshold:
    listx.append(i)

restart = 0
restartLoop = True
while restartLoop:
    restartLoop = False
    for idx, i in enumerate(listx):
        print("do something as printing i:", i)

        if i > 5:  # if this condition: remove points and restart loop to search again in range
            print("found value for condition: ", i)
            del listx[idx]  # simulates removing points for a certain i value

            #  optimization would happen here

            restartLoop = True
            print("RESTARTING LOOP\n")
            restart += 1
            break  # break inner while and restart for loop
        else:
            # continue if the inner loop wasn't broken
            continue
    else:
        continue

print("restart - outer while", restart)
Looking forward to see your new version of the code!

[mks_gis]

Hi Luiz,

 Thanks for your code. Not been able to do much on this lately, and to go through your code properly, but have updated to accommodate Metashape and some minor tweaks. Thought I'd share until I have more time.

Cheers
Martin

[fjgarciam]

Hi Luiz,

 Thanks for your code. Not been able to do much on this lately, and to go through your code properly, but have updated to accommodate Metashape and some minor tweaks. Thought I'd share until I have more time.

Cheers
Martin

Hi the script is updated? i have problems with the variable Hightaccurazy

[Paulo]

Hi fjgarciam,

i updated the script to v8 to be compatible with MS v1.6 upwards.... Basically matchPhotos uses downscale parameter instead of accuracy and buildDepthMaps uses also parameter downscale instead of quaity...

Aso introduced 2 boolean parameters (gen_pre and ref_pre) to use to define preselection values (T or F) in matchPhotos process...

Also changed filter parameter of buildDepthMaps process to filter_mode

There  maybe other changes necessary for builModels, dem and ortho but look if it works for you....

[mks_gis]

Thanks Paul, I'll look at your version and perhaps merge them.

The maintained version is on Github:
https://github.com/gisportsmouth/PhotoScan-Automation-Script