Forum

Author Topic: Using external drives (USB)  (Read 2561 times)

Udjat

  • Newbie
  • *
  • Posts: 5
    • View Profile
Using external drives (USB)
« on: January 31, 2022, 10:48:34 PM »
Hi there,

I'm recently handling projects which create really big files (up to TBs) and also they require a huge amount of pictures. Right now I'm using Windows 11 on an SSD drive and I have a couple of built-in regular hard drives. I use one of those for the pictures and the other one for the metashape project files. So, as I'm running out of storage capacity quite fast, here they are my questions:
1.- Does loading pictures from external USB hard drives slow down the process?
2.- Does processing Metashape projects on external USB hard drives slow the process (compared to built-in hard drives)?
3.- Is it better to load pictures from one drive and write the project files on a different one or to do load the pictures and process the project on the same drive? (either built-in or external USB drives)
4.- Does the use of SSD drives improve the processing speed on any way?

Thanks a lot!

Bzuco

  • Full Member
  • ***
  • Posts: 178
    • View Profile
Re: Using external drives (USB)
« Reply #1 on: February 01, 2022, 12:41:00 AM »
You could see maybe some slowness during detecting points phase if you have more than one GPU card(e.g. 3x RTX 3070 or better) and all are feeded during this phase from classic HDD drive with read speeds ~ 100MB/s.
I am using RTX 2060 super and 18Mpix JPEG files, each ~ 10MB big. My system can feed GPU at speed ~ 3-4 JPEGs/s and my disk read speed during that phase is ~30-35 MB/s...even if I copy that files on SSD, the read speeds are the same.
USB 3.0 connection speeds are much much higher than it is necessary.
You will need good write speed at the end of dense cloud creation phase when partial dense clouds are merged into final one...you can check read/write speed in task manager performance tab during this process. If you will see constantly read or write speeds higher than ~100-130MB/s then it would be worth to save project files on SSD...otherwise classic HDD will be enough.
You can have photos and project files on the same drive, because after adding photos into empty project and start aligning phase, the photos will be automaticaly cached in system RAM by windows.

jenkinsm

  • Jr. Member
  • **
  • Posts: 72
    • View Profile
Re: Using external drives (USB)
« Reply #2 on: February 01, 2022, 06:36:22 AM »
You could see maybe some slowness during detecting points phase if you have more than one GPU card(e.g. 3x RTX 3070 or better) and all are feeded during this phase from classic HDD drive with read speeds ~ 100MB/s.
I am using RTX 2060 super and 18Mpix JPEG files, each ~ 10MB big. My system can feed GPU at speed ~ 3-4 JPEGs/s and my disk read speed during that phase is ~30-35 MB/s...even if I copy that files on SSD, the read speeds are the same.
USB 3.0 connection speeds are much much higher than it is necessary.
You will need good write speed at the end of dense cloud creation phase when partial dense clouds are merged into final one...you can check read/write speed in task manager performance tab during this process. If you will see constantly read or write speeds higher than ~100-130MB/s then it would be worth to save project files on SSD...otherwise classic HDD will be enough.
You can have photos and project files on the same drive, because after adding photos into empty project and start aligning phase, the photos will be automaticaly cached in system RAM by windows.


You, sir, are a veritable goldmine of information! Do you teach a course on what you do? Or have any tutorials online? If not you should consider it!

Bzuco

  • Full Member
  • ***
  • Posts: 178
    • View Profile
Re: Using external drives (USB)
« Reply #3 on: February 01, 2022, 01:13:57 PM »
@jenkinsm
Thank you for your kind words. Sadly no tutorials, I'm glad I'm able to translate my knowledge from my language into English on forums  :)

Udjat

  • Newbie
  • *
  • Posts: 5
    • View Profile
Re: Using external drives (USB)
« Reply #4 on: February 01, 2022, 08:49:15 PM »
Thank you very much @bzuco! you helped a lot!

SimonBrown

  • Jr. Member
  • **
  • Posts: 62
    • View Profile
    • AccuPixel Ltd - Dealer and Training Centre
Re: Using external drives (USB)
« Reply #5 on: February 02, 2022, 12:18:56 PM »
3.- Is it better to load pictures from one drive and write the project files on a different one or to do load the pictures and process the project on the same drive? (either built-in or external USB drives)
4.- Does the use of SSD drives improve the processing speed on any way?


Occasionally I forget to copy source images from a network share to the local SSD.

Things slow down you could say...but compared with other steps in the overall workflow its not hugely significant.

Its been so long since I used a mechanical as opposed to SSD for processing I cannot remember, but read/write speed isn't the greater bottleneck.

Processor speed and availability (or not) of GPU would be applying the limits.

Agisoft endorsed online Metashape training - see: https://accupixel.co.uk/metashape-training-courses

Costas

  • Jr. Member
  • **
  • Posts: 67
  • Aerial Mapping
    • View Profile
Re: Using external drives (USB)
« Reply #6 on: February 02, 2022, 02:25:51 PM »
So, as I'm running out of storage capacity quite fast, here they are my questions:

Metashape indeed creates huge amount of data. (as you can see here -> https://www.agisoft.com/forum/index.php?topic=13771.0 )

One thing you can do to save huge amount of space is to "Remove Orthophotos" assuming that you do not need to do some patching after.

Udjat

  • Newbie
  • *
  • Posts: 5
    • View Profile
Re: Using external drives (USB)
« Reply #7 on: February 07, 2022, 04:41:23 PM »
@SimonBrown Thank you very much! I just started using USB drives to expand my storage capacity but I really wondered if it was a bad decision. Thanks again!

@Costas That's a GREAT tip!! Now it makes sense the "remove ortophotos" option, thanks a lot!

CheeseAndJamSandwich

  • Full Member
  • ***
  • Posts: 153
    • View Profile
    • Sketchfab Models
Re: Using external drives (USB)
« Reply #8 on: February 07, 2022, 05:42:28 PM »
You can have photos and project files on the same drive, because after adding photos into empty project and start aligning phase, the photos will be automaticaly cached in system RAM by windows.
I definitely notice this before i even add the photos to MS!
I download them off the SD Card to my SSD, and then immediately backup the SD Card to an external HDD...  That 2nd copy operation takes a fraction of the time, as all of the 3,000 jpgs are still in Windows' cache!
Here it pays to have 'too much' RAM (64GB in my case)...  Even if the applications themselves don't get faster if they've already got enough, as Windows will just use all those 'excess' GB or RAM for caching lots and lots of files.

Oh, and the other obvious point to be wary of:  Make sure the cable can't be knocked, such that the External HDD might disconnect/reconnect momentarily (Laptop users!)...  Sod's law dictates that this would certainly happen right at the end of the processing job as it's writing out the finished data!