Progress report from python

[sven_a]

We are working with very large datasets that we are processing in the background with Agisoft. To get a time/progress estimate we would like to write this information to a file that we can open. To limit to amount of write actions and to keep a somewhat managble overview we would like to report the progress every 10% or 10 minutes. We are working with this code (somewhat adapted from a few lines of code found on this forum):

Code: [Select]

def progress_print(p):
    """
    MAKE COPY OF FILE BEFORE OPENING DURING ACTIVE RUN!
    :param p:
    :return:
    """
    global last_progress_percentage
    global last_progress_minutes

    elapsed = float(time.time() - start_time_processing_step)
    elapsed_min = elapsed / 60
    elapsed_progress = p - last_progress_percentage

    if p and (elapsed_progress > 10 or elapsed_min > 10):
        last_progress_minutes = elapsed_min
        last_progress_percentage = p
        sec = elapsed / p * (100 - p)
        hhmmss = datetime.timedelta(seconds=int(sec))

        with open(progress_status_path, "a") as file:
            dt_string = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
            progress_text = "{}; step: {} ({}/{}); progress: {:.2f}%; estimated time left (HH::MM:SS): {}\n".format(
                dt_string, processing_steps[processing_steps_counter], processing_steps_counter + 1,
                len(processing_steps), p, hhmmss)
            file.write(progress_text)
The issues is that the global variables do not seem to always update correctly (but weirdly sometimes it is updated) and consequently this code is triggered way too often (especially in the build depth maps and generate dense cloud steps). Any idea how we can succesfully update the variables last_progress_percentage & last_progress_minutes. Now it does not always update correctly. Any ideas or help would be really appreciated.

[Alexey Pasumansky]

Hello sven_a,

Do you have issues with the 10 minutes check or there's also a problem with 10% step update?

[sven_a]

Both. But the issue is clearly visible in the 10% step update.

In the attached image (previous post) the printed lines should only trigger when there has been a progress increase of 10% or more. But as can be seen the last_progress_percentage is in both lines approx. 60.41. Consequently P-Last Progress is the second time still > 10%. While if it worked as intended the last_progress_percentage should have updated to 75.26. Last_progress_min is also not updated.

Thank you! 

[Alexey Pasumansky]

Hello sven_a,

For me it looks like last_progress_minutes is not properly used in your code. The script is always checking that elapsed_min is bigger than 10 - which means that it is true always after 10 minutes from the processing stage start.

Here's a sample code that I've used, basing on your script, that should update the file contents either after 10% progress is passed or 10 minutes since the last update (or script start). For debugging purposes you can lower the threshold for percentage and raise for minutes or vice versa to check the behavior of any parameter.

Code: [Select]

import Metashape, time

def progress_print(p):
global start_time_processing_step
global last_progress_percentage
global last_progress_minutes
global file

elapsed = float(time.time() - start_time_processing_step)
elapsed_min = elapsed / 60 - last_progress_minutes
elapsed_progress = p - last_progress_percentage

if p and (elapsed_progress > 10 or elapsed_min > 10):
last_progress_minutes = elapsed_min
last_progress_percentage = p

file.write("{:.1f}%\t{:.2f} sec elapsed,\t{:.1f} min last\t{:.1f}% last\n".format(p, elapsed, last_progress_minutes, last_progress_percentage))
file.flush()


chunk = Metashape.app.document.chunk
global start_time_processing_step
global last_progress_percentage
global last_progress_minutes
global file
last_progress_percentage = 0
last_progress_minutes = 0
start_time_processing_step = time.time()
file = open("D:/progress.txt", "wt")
chunk.matchPhotos(accuracy = Metashape.LowAccuracy, progress=progress_print)
file.close()