Forum

Author Topic: Is there any way to speed up importing a large number of images?  (Read 2027 times)

jenkinsm

  • Jr. Member
  • **
  • Posts: 72
    • View Profile
I am working on several gigantic projects with well over 200,000 TIFF images each. I would like to know if there are any tips or hidden secrets that would allow me to import all of my photos much faster.

The slow importing happens on my Windows machine where the photos are stored on a 32 TB RAID 0 (2 drives), whereas my M1 Max MacBook Pro loads photos from an external SSD almost instantly. Granted, I haven't tried 100k at once on the MacBook, and I know it's faster partly because of the SSD, but I'm still surprised by how long it takes on the PC.

Putting the photos on an SSD is not an option, as each project uses several TB of photos. Converting to PNG or JPEG is not ideal because that's an extra step I don't want to include, and my workflow can only output TIFF/DPX/EXR frames directly (I'm capturing roads on video and using DaVinci Resolve to mask out the sky, adjust the contrast/midtone detail, and mask out the capture vehicle).

Any tips or tricks I should try?
« Last Edit: March 30, 2022, 04:33:18 AM by jenkinsm »

Bzuco

  • Full Member
  • ***
  • Posts: 195
    • View Profile
Re: Is there any way to speed up importing a large number of images?
« Reply #1 on: March 31, 2022, 09:07:10 PM »
If I understand, your output is TIFF image sequence from DaVinci Resolve and then you add all those 200.000 files into metashape project?
How big are those TIFF files?

jenkinsm

  • Jr. Member
  • **
  • Posts: 72
    • View Profile
Re: Is there any way to speed up importing a large number of images?
« Reply #2 on: April 02, 2022, 10:17:04 PM »
If I understand, your output is TIFF image sequence from DaVinci Resolve and then you add all those 200.000 files into metashape project?
How big are those TIFF files?

Yes, that is correct. Currently, I am using 8-bit compressed TIFF images which are 13.6 MB each. In the future, I plan to use a better camera to capture with, which would warrant exporting 16-bit TIFF images which are 86.7 MB (I think that's for uncompressed files).

Bzuco

  • Full Member
  • ***
  • Posts: 195
    • View Profile
Re: Is there any way to speed up importing a large number of images?
« Reply #3 on: April 03, 2022, 12:53:36 AM »
I made some test with 262 jpeg files(18Mpix) and the fastest import was with JPEG files 21s, then 34s TIFF(none compression), then 47s BMP format, then 58s TIFF(JPEG compression), then 90s TIFF(LZW compression)...so for you tiff with none compression should by the fastest way. I did not try exr format.

You can run several instances of metashape, in each instance import different sequence of images, save project in each metashape instance, then choice one instance and append other saved projects files as chunks, then merge chunks into one and you are done  :).
Importing process is single threaded operation, so this way you can linearly increase speed and utilize more than one CPU core. Another option is kindly ask devs, if it is possible to make import procedure multithreaded  ;D
Soon you will be probably limited by your raid system, I am assuming that you have 2x 16TB disks and each has maximum read speed of ~ 250-260MB/s on the begining, so more than 500MB/s you will be not able to achieve. With more non raid physical disk your import speed could be faster, e.g. 8x 4TB disks ~ 8x 150MB/s....but that speed can be achieved only from 0 - ~50% of disk capacity, then speed drops.