Bug loading photos

[mrv2020]

Hello,

We individually analyze each of the aligned images on a daily basis and make notes using the point of shapes. We need to analyze more than 5000 photos and the metashape has a delay to load the photos, in version 1.6 this did not happen, but in the new versions each time it presents an increase in the delay in opening the individual images, we see the thumbsnails and mask with the message from bottom right side loading 0... 100%. This loading takes 2 to 3 seconds generates a delay for us to analyze more than 2 to 3 hours per unproductive day for 5000 photos. We have already left the show depth maps and mask button disabled and still nothing is resolved.

There is a mention of improvements in the post below but no news so far.
https://www.agisoft.com/forum/index.php?topic=13298.msg61409#msg61409

I ask, as far as possible, that you look fondly at this topic and bring a long-awaited solution for some time.

Best Regards.

[mrv2020]

Hi Alexey, look at us again here.

I would like to report slowness for large groups of images, above 3-4k images per chunk.

We are working with very large projects more than 150k images per section for a total of 4 going up to 5 tb so far and growing. It takes a few hours to just import the 150k. If "Show Camera" is active, you can leave and come back tomorrow. To be able to process, we reduce it to 3-5k images per chunk. And to select we have to see the macro, deactivate the show camera, manipulate the zoom and turn it off again. Selecting is another pain, it's a good thing it has the circular selection because the others simply lock and you have to keep your finger pressed during the selection for minutes.

However, to select the necessary images by coordinates, I need to open and load them all in a single chunk. After saving, I use CMD to copy the folders because trying to save with another name takes a long time. Then I copy it via CMD, open the new Chunk project and delete it until I reach 3-5k. We are using M2 units, Xeon Servers, 128gb Memories and Dual RTX2080. In the Windows Explorer manager we can open it in the viewer and see the final format instantly.

In older versions, prior to those that loaded the depth map, this did not happen and since then we have only had slowness problems. Too bad these versions have bugs.

Extracting information in these large chunks with the show camera active plus the time it takes to load individual images by loading Exif or thumbs is making it a mental challenge of patience to work with these projects.

Please help those poor nocturnal coffee guzzling users to get one night's sleep lol.

Best regards,

[Alexey Pasumansky]

Hello mrv2020,

What image format you are typically using, what options are enabled in Import/Export section of the Advanced Preferences tab and if the images are stored on local, network or external drive? If it is not a local drive, then what is the communication speed with the device?

[mrv2020]

Hello Alexey,

They look like this:

Project Files
*Disable maintain key and depth maps
* Try On/Off absolute image path, and without gains,

Export/import
*Try thumbnails On/Off Load, and without gains,
*Disable all payload calibrations/orientation/location/GPS/Satellite RPC

Miscellaneous
*Enable VBO
*Others Off


And to process we use the project and images on local drives, the PC works with three M2 Samgung drives with 2tb.

If you use network storage the time doubles or more.

Chunks with 2k or less images are not happening.

[tazzo]

Are .tiff, .png, .jpg? Can you upload a file sample?

[mrv2020]

Sorry, we use camera-style frames, with standard Sony compression, 24 megapixel images, JPG format

[tazzo]

Have you tried to disable Windows Defender during the processing?
Running a script that remove unused metadata may help to speed-up loading time, but you need to do some testing do understand what metadata can be stripped out to speed-up things.

[mrv2020]

Yes, I tested the possibility of Windows Defender being harmful and it is not.

It was one of the first lines we followed, trying to clean up unnecessary metadata, but it didn't increase speed.

The best we can do is reduce the chunks to 3 to 5k images.

[tazzo]

You can try to buy and install a fast .jpg decoder codec, I think that the default Windows one is not super performant..

[Alexey Pasumansky]

Hello mrv2020,

Thank you for additional information.

Would be actually interesting to see, if the same time is spent for loading photos stage, if you use a different computer in the same conditions (same type of data access, same amount of images).

[mrv2020]

Perfect,

In the loading of 56,793 images were spent
 216.853 sec -  i9 windows 11
 212.221 sec -  Xeon E5-2696v3 Windows 10


What we find out, folders can not have more than 4300 files, even Windows quickly opening the thumbs.

In these folders we can, such as IrfanView navigating all images and not thumbs quickly.

[tazzo]

I think that Windows is loading thumbnails of the folder view only, if your monitor is displaying 20 thumbs, only 20 are loaded. Metashape is actually loading all maybe.

[mrv2020]

Alexey and Tazzo, Firstly, thank you for your attention, I will try to improve the description and point out where the bottlenecks occur.

What I'm referring to is, when I open a single photo in viewers, such as irfanview, the photo opens full instantly, I rotate the entire folder image by image, and all the images open full instantly.

Where the slowdown occurs is:

In the Model or Ortho tab
At this stage, we are never opening the images to see their content, and Show Thumbs/Slaves/Local Axes will be turned off.

What is the slowness like?
At this stage we only import the images, the initial import delay is tolerable due to the volume.

After importing, a simple selection of images in Model or Ortho becomes something impossible to do with more than 50k images, they take several seconds to be able to select, delete or just drag and move the screen.

But we need to put them all together to separate the groups by coordinates. It takes a lot of time but we managed to separate and reduce 50k to chunks with up to 5k images. This is really terrible because we need to leave the "Show Cameras" function active at this stage.

With 5k we follow the flows, align, generate dense cloud, DEM and Orthophoto.

After finishing, we open the Ortho tab and edit the shapes. If the "Show Camera" is active, we cannot navigate the ortho, if we create a polyline shape, the delay takes several seconds to click, respond and move.

By deactivating the "Show Camera", the delay disappears and we can create and edit the shapes on the orthophoto smoothly until reaching the 1.6mb limit, the shape.zip file above that starts to gradually slow down.

In the Photos Tab
Well, after editing the shapes in the Ortho tab, we open image by image for annotations and reviews. At this point, when browsing image by image, the message (Loading...) appears in the bottom corner of the Photo tab.

The Thumb of the image, either Exif or the thumb.zip file that metashape uses, loads instantly, so the Exif of the Advanced tab on or off makes no difference. The delay takes place after the Thumb loads.

Therefore, the program instantly opens the Thumb of the image and then opens the image file. And this process of loading the image (Loading...) It takes 2-3s to read the contents of the image file on the M2 drive.

We have already tested with images reduced to 12mpixel and larger to 42mpixel, the delay is almost always the same.

[rossnixon]

I see you want to "But we need to put them all together to separate the groups by coordinates".

You can sort images by either latitude and longitude by exposing the EXIF data in Windows File Explorer details view.
First you need to do a registry tweak to allow those columns to appear.

I have attached a .reg file which allows this to happen.
You can merge this into the Windows Registry.
Recommended:  Preview .reg files in NotePad to see what they are about to do.

See screenshot from my computer after using this:

[mrv2020]

Cool, thanks for sharing Rossnixon,

The project involves daily inspections in areas with and without GPS coverage, more than 80% have precise coordinates, 18% approximate and the rest do not.

So, depending on it, I need to mix several days to compose the scenario. For this reason we try to use a script to filter the images within a polygon and then refine it to reach the limit.

When it comes to extracting the values, instead of Windows we use exiftools, which is quick and practical to export the lat long values, among others, from the exifs directly to a csv.