Forum

Author Topic: Optimizing Processing Time?  (Read 6271 times)

Shovelhead

  • Jr. Member
  • **
  • Posts: 75
    • View Profile
Optimizing Processing Time?
« on: March 04, 2023, 09:24:35 PM »
I recently discovered the benefits from processing my point cloud using higher quality settings when constructing my point clouds. However I would like to increase the time it take to process.

I have a dedicated workstation I use for my photogrammetry processing with the following specs;

NVIDIA GeForce GTX 1070
i7 - 7700 CPU @ 3.60GHz
64.0 GB of Ram
Windows 10

I'm fairly ignorant when it comes with computer hardware and have two questions.

1. How can I make sure Agisoft is using what I have to the fullest to ensure the fastest processing time possible?
2. What would my best "bang for my buck" be to increase processing time? Ex. A better video card, more RAM, ect.

My recent project took 3 days 19 hours for depth maps generation time and 2 days 4 hours for dense cloud generation time. Would it be possible to get both of these combined under 24 hours without spending $1000 and retaining the quality?

jenkinsm

  • Jr. Member
  • **
  • Posts: 74
    • View Profile
Re: Optimizing Processing Time?
« Reply #1 on: March 05, 2023, 06:07:11 AM »
A project like yours would take a few hours at most on modern hardware. CPU speed is extremely important here, followed by GPU and RAM speed. RAM quantity matters but doesn't affect speed much. More RAM allows you to create large projects.

Metashape automatically uses your hardware "to the fullest" - if it's not, then there is a bottleneck somewhere. A common bottleneck is your storage device. Use an SSD, not a hard drive, to store your project images. Otherwise, the hard drive is likely to be a bottleneck. You can check this in Task Manager - if the drive with your images is at 100% usage, then it's a bottleneck

If you're talking about upgrading your existing components, it's probably better to build a completely new PC. You should get the CPU with the fastest clock speed that you can afford. A 13900KS would be at the top of my list for a new PC, but that's $700 for the CPU alone. So instead you might want to look at an older CPU like a 10900K that can run at or above 5 GHz (that's what I have). You would need a powerful water cooler to achieve 5+ GHz full-time, but it blazes through photogrammetry projects like nobody's business. It's a lot faster than a 4.5 GHz Xeon with 48 threads, even though it only has 20 threads.

If you are not ready to upgrade, you should overclock your existing CPU as much as possible. That will help speed up your processing and might only cost as much as a good air cooler or AIO water cooling loop.

The GTX 1070 is OK for making depth maps but it has little to no impact on creating a dense cloud since that happens on the CPU.

Lastly, do you really need the point cloud in the first place? You should try making your mesh using depth maps only and see if the results are satisfactory. That's a much faster process to begin with and would take a lot less time to create a mesh.
« Last Edit: March 05, 2023, 06:11:22 AM by jenkinsm »

jenkinsm

  • Jr. Member
  • **
  • Posts: 74
    • View Profile
Re: Optimizing Processing Time?
« Reply #2 on: March 05, 2023, 06:10:16 AM »
A project like yours would take a few hours at most on modern hardware. CPU speed is extremely important here, followed by GPU and RAM speed. RAM quantity matters but doesn't affect speed much. More RAM allows you to create large projects.

Metashape automatically uses your hardware "to the fullest" - if it's not, then there is a bottleneck somewhere. A common bottleneck is your storage device. Use an SSD, not a hard drive, to store your project images. Otherwise, the hard drive will likely be a bottleneck. You can check this in Task Manager - if the drive with your images is at 100% usage, then it's a bottleneck

If you're talking about upgrading your existing components, it's probably better to build a completely new PC. You should get the CPU with the fastest clock speed that you can afford. A 13900KS would be at the top of my list for a new PC, but that's $700 for the CPU alone. So instead you might want to look at an older CPU like a 10900K that can run at or above 5 GHz (that's what I have). You would need a powerful water cooler to achieve 5+ GHz full-time, but it blazes through photogrammetry projects like nobody's business. It's a lot faster than a 4.5 GHz Xeon with 48 threads, even though it only has 20 threads.

If you are not ready to upgrade, you should overclock your existing CPU as much as possible. That will help speed up your processing and might only cost as much as a good air cooler or AIO water cooling loop.

The GTX 1070 is OK for making depth maps but it has little to no impact on creating a dense cloud since that happens on the CPU. Upgrading your GPU will help speed up the depth map generation and image matching, but not much else.

Lastly, do you really need the point cloud in the first place? You should try making your mesh using depth maps only and see if the results are satisfactory. That's a much faster process to begin with and would take a lot less time to create a mesh.

tazzo

  • Full Member
  • ***
  • Posts: 159
    • View Profile
Re: Optimizing Processing Time?
« Reply #3 on: March 05, 2023, 04:23:17 PM »
To optimize processing time, my advices are:
- be sure to remove eccessive camera overlap
- use 8 bit images (eventually tonemapped from 16 bit images)
- use markers (automatically detected or manually placed)
- use accurate camera calibration profiles

Helping the algorithm by providing some additional data (marker, calibration, etc..) and removing redudancy  is a good start to reduce unuseful processing time.

To optimize the GPU ram you can set the monitor to a lower resolution and/or use the integrated graphics to render the desktop instead of the dedicated GPU. Disabling the antivirus / unuseful processes will help a little as well if you are running low on RAM.

Corensia

  • Jr. Member
  • **
  • Posts: 54
    • View Profile
Re: Optimizing Processing Time?
« Reply #4 on: March 09, 2023, 06:12:41 AM »
So with your budget of under $1000, I would recommend the following upgrades.

CPU: AMD 5800X - $220 MSRP   #Extremely cost-efficient cpu but uses an AM4 motherboard which means you won't be able to upgrade to an AM5 CPU in the future, also might be hard to find
Motherboard: Any AM4 MB - $100
GPU: AMD 6800 XT - $560 MSRP   #Huge performance boost from the 1070
RAM: Use your old 64GB - $0
PSU: ~800W at least - $100

Total : $980

Pretty sure this setup is the most cost-effective Metashape performance you'll get under $1000

tazzo

  • Full Member
  • ***
  • Posts: 159
    • View Profile
Re: Optimizing Processing Time?
« Reply #5 on: March 09, 2023, 12:42:35 PM »
You can also consider an external GPU with a lot of memory (ex. RTX3060 12 GB) if you have a thunderbolt 3 connection.

Shovelhead

  • Jr. Member
  • **
  • Posts: 75
    • View Profile
Re: Optimizing Processing Time?
« Reply #6 on: March 09, 2023, 10:42:17 PM »
Thank you very much for all the replies. This is a great jumping off point to dig into my research further. Very much appreciated.

I started processing a RTK flight yesterday (my first) and I'm getting that it will take 655 hours to estimate camera positions. I have about 2000 images at 20 MP each and set my accuracy to high.

I used the new mavic 3 enterprise with the smart oblique option and didn't realize how many images it would take. Any idea how I can get this down to a reasonable time? I guess my only option is lowering the accuracy?

tazzo

  • Full Member
  • ***
  • Posts: 159
    • View Profile
Re: Optimizing Processing Time?
« Reply #7 on: March 09, 2023, 11:16:01 PM »
Is the GPU checked under Tools / Preferences / GPU?

Shovelhead

  • Jr. Member
  • **
  • Posts: 75
    • View Profile
Re: Optimizing Processing Time?
« Reply #8 on: March 10, 2023, 12:13:13 AM »
I actually just checked it this time, I'm guessing that was a mistake?

tazzo

  • Full Member
  • ***
  • Posts: 159
    • View Profile
Re: Optimizing Processing Time?
« Reply #9 on: March 10, 2023, 12:32:28 AM »
Can you try to align a subset of the flight (ex. 50 images) with and without the GPU and report processing time of both cases?

olihar

  • Full Member
  • ***
  • Posts: 185
    • View Profile
Re: Optimizing Processing Time?
« Reply #10 on: March 10, 2023, 01:44:13 AM »
I can run this dataset for you and report on the time if you want.

Shovelhead

  • Jr. Member
  • **
  • Posts: 75
    • View Profile
Re: Optimizing Processing Time?
« Reply #11 on: March 10, 2023, 02:15:01 AM »
I checked it off and dropped the accuracy to medium and it seems to be going a lot faster. Im not at estimating camera positions yet.

Here is the dataset if anyone wants to try it out... https://1drv.ms/f/s!AkXoacH8d5pA1E5cvxS2BEfVoS8n?e=4k1Jtp

Shovelhead

  • Jr. Member
  • **
  • Posts: 75
    • View Profile
Re: Optimizing Processing Time?
« Reply #12 on: March 10, 2023, 02:15:25 AM »
I can run this dataset for you and report on the time if you want.

That would be amazing, thanks.

Paulo

  • Hero Member
  • *****
  • Posts: 1350
    • View Profile
Re: Optimizing Processing Time?
« Reply #13 on: March 10, 2023, 07:54:20 AM »
Hello Shovelhead,

 aligned a 100 image subset on a much less powerful system than yours (CPU i7 3690Q  at 2.4 GHZ, 16 GB Ram and Nvidia GTX 670M graphic card) and alignment was done in 2 min so on your system with 6000 images you should have no problem aligning ina reasonable time. Note that last part of alignment (estimating camera positions) takes up much less time than matching images....
Best Regards,
Paul Pelletier,
Surveyor

Shovelhead

  • Jr. Member
  • **
  • Posts: 75
    • View Profile
Re: Optimizing Processing Time?
« Reply #14 on: March 10, 2023, 08:12:38 AM »
It went much faster with me this time with that box unchecked. What exactly is it doing with it checked?

Also, I cant seem to import the accuracy of the images automatically so I forced them to 0.020m and after alignment I'm seeing large error (around 0.40m). I must be doing something wrong here? The images do contained accurate location metadata so I assumed if I forced their accuracy to a reasonable level it would process out.