Forum

Author Topic: Processing Time - Build Point Cloud  (Read 859 times)

Tas

  • Newbie
  • *
  • Posts: 48
    • View Profile
Processing Time - Build Point Cloud
« on: October 31, 2024, 02:04:21 PM »
Hi All,
This is a time-sensitive inquiry, and I'm hoping that someone can provide input or point me in the right direction.

I'm trying to...
  • Confirm whether my current processing step will ever be completed (or if I should restart depth map and point cloud generation with lower settings).
  • Determine what's causing processing time to skyrocket on some of my photogrammetry projects. I've invested in a relatively powerful PC, and it doesn't seem like hardware is the issue for a project of this size.

Project Info:
  • 2924 photos (.jpg), most are 45MP
  • 69 colored laser scans (.e57)
  • Agisoft Metashape Version 2.1.2 build 18548
  • Model subject:  masonry building exterior, with trees/vegetation

PC Info
  • CPU:  AMD Ryzen Threadripper Pro 7975WX (4.0GHz 32 Core 350W), sitting at ~11% utilization
  • GPU:  NVIDIA GeForce RTX 4090, ~22-23GB free memory referenced in Metashape processing logs, sitting at 1-2% utilization
  • RAM:  512 GB DDR5 (8x Kingston DDR5-5600 ECC Reg. 2R 64GB), sitting at ~13% utilization
  • Processing Drive:  SSD with >1TB free space (Sabrent 8TB Rocket 4 Plus Gen4 PCIe NVMe M.2). All files are saved locally.

Processing in Progress
  • Depth map generation completed with High quality and Mild filtering settings
  • Stuck on "Generating Point Cloud...", with processing time estimates continuing to increase. ~11 hours elapsed so far
  • Current repetitive and time-intensive processing steps include "preloading data" (typically 1-11 minutes) and "preprocessing images" (typically 0.5-2 minutes)"
  • Processing completion percentage has increased by perhaps 1% in the last 1.5 hours, and the estimated remaining time has increased by ~10 minutes in that timeframe.
  • The in-progress processing log is too large to attach but can be downloaded or viewed here: https://drive.google.com/file/d/1MxPVx3U69fnsf6SAYEUHuBxUZ0zR1Yoc/view?usp=sharing


Thank you!

vineg

  • Newbie
  • *
  • Posts: 29
    • View Profile
Re: Processing Time - Build Point Cloud
« Reply #1 on: October 31, 2024, 04:39:00 PM »
Hi!

Please, try 2.2.0 pre-release from the topic https://www.agisoft.com/forum/index.php?topic=16745.0
There was significant performance improvements when building point cloud in a project with laser scans.

You can probably cancel processing and save the project to keep depth maps so you later can reuse them.

Building point cloud with 2.2.0 should probably reduce processing time for point cloud step of this project from ~15 hours to ~2 hours.

Regards, Egor
« Last Edit: October 31, 2024, 05:00:31 PM by vineg »

Tas

  • Newbie
  • *
  • Posts: 48
    • View Profile
Re: Processing Time - Build Point Cloud
« Reply #2 on: October 31, 2024, 05:39:11 PM »
Thanks for the response, vineg!

I'm now running the 2.2.0 pre-release, even though this is a professional deliverable. I'm hoping that I don't run into any surprises for other steps in the workflow. If I do encounter issues, do you know if it is safe to say that I'll be able to "roll back" to version 2.1.3 and continue processing the mesh, orthomosaic images, etc.?

FYI, I'm currently running two instances of metashape (2.2.0) to generate the point cloud for this project. One instance is using High quality depth maps, and the second instance is using medium quality. Both are at the point where the log is "filtering ~photo name~ using 100 neighbors...", and the estimated remaining duration is 4h (high quality) and 3.5h (medium quality).


vineg

  • Newbie
  • *
  • Posts: 29
    • View Profile
Re: Processing Time - Build Point Cloud
« Reply #3 on: October 31, 2024, 06:58:21 PM »
There is a note in 2.2 topic:

Project files created with 2.2.0 version can't be opened with Metashape 2.1 and earlier versions.

So it is not safe to say, but it typically should work. I can only remember, that multiple sets of masks may break compatibility.

Regards, Egor
« Last Edit: October 31, 2024, 07:23:18 PM by vineg »

Tas

  • Newbie
  • *
  • Posts: 48
    • View Profile
Re: Processing Time - Build Point Cloud
« Reply #4 on: October 31, 2024, 08:13:21 PM »
Thanks, Egor.

Unfortunately, I do seem to be running into the same issue where processing comes to a relative standstill when the point cloud generation process reaches the "preloading data" step in the processing log. Even when running just the medium-quality version, the first "preloading data" step took ~12 minutes, and the first "preprocessing images" step took 2-3 minutes. The estimated processing time is also climbing again.

Any thoughts?

Bzuco

  • Full Member
  • ***
  • Posts: 225
    • View Profile
Re: Processing Time - Build Point Cloud
« Reply #5 on: November 01, 2024, 10:50:52 AM »
Hi Tas, you should definitely lower your depth maps settings, because at the end of your log Metashape already extracted  1 059 126 305 points(which is a lot lot) and that is not the end.
Try to guess how many millimeters is covering one pixel on your 45 Mpix photo and decide if you need such a precision of your final point cloud.

vineg

  • Newbie
  • *
  • Posts: 29
    • View Profile
Re: Processing Time - Build Point Cloud
« Reply #6 on: November 01, 2024, 02:31:50 PM »
In 2.2.0 "preloading data" stage was optimized to use depth maps for laser scans, if available. In 2.1 laser scans depth maps were not calculated, so it is required to rebuild depth maps. Sorry for missing that. This stage should not normally exceed 30 sec with usual ssd (you could check load just in case). Also I missed that extracting points is takes many time as well, so my ~2h estimate is really off.

Reagards, Egor

Tas

  • Newbie
  • *
  • Posts: 48
    • View Profile
Re: Processing Time - Build Point Cloud
« Reply #7 on: November 01, 2024, 05:50:37 PM »
I can't thank you both enough for taking a look at this. I really, really appreciate your time.

I'm still finding that the Build Point Cloud step takes a very long time to generate even a small area of the point cloud. E.g., it'll take an hour to generate a point cloud when the bounding box is set to just a 5' x 5' area.  I'm running multiple iterations to try to reconcile misalignments between the photogrammetry-generated point cloud and the laser scans. I'm now past my project deadline, so there's some time pressure. After finding an apparent bug in 2.2.0 that won't allow me to process new depth maps, I'm now resetting the alignment for the entire project in a last-ditched effort. I'll, of course, report the bug in the relevant thread.

Egor, please excuse my ignorance, but can you clarify what you mean by "check load?" The percentage of time the drive is processing read/write requests (as graphed in the task manager performance window) is very low. I've checked the read-write speeds of the processing drive and got Read = 5324 MB/s and Write = 5301 MB/s. There seems to be a significant disconnect between that read time and the "preloading data" time. E.g., in one example, it took ~559 seconds to load ~5121 MB of depth maps -- it seems like it should've taken less than a second. Am I missing something here?

For anyone who comes across this thread in the future, I checked read/write speeds by following this brief tutorial:  https://www.youtube.com/watch?v=mjcGfnVdqz4


Bzuco, I've abandoned the high-quality settings and am working with medium settings. I'm very familiar with the process of GSD and detectable feature size in relation to the photo resolution/sensor and capture distance. I'm not familiar with choosing my depth map quality settings to correlate to the desired detectable feature size, though. Do you know of a way to correlate the depth map settings to a desired point cloud "resolution"? Or perhaps a relevant resource that I've missed? Lastly, I'd love to understand where you say the total extracted points in the log file - I couldn't find your referenced value (1 059 126 305) anywhere in the log file, and I thought the log only gave you a total for points generated from a subset of depth maps.

Thanks for reading and any additional input.


Tas

  • Newbie
  • *
  • Posts: 48
    • View Profile
Re: Processing Time - Build Point Cloud
« Reply #8 on: November 01, 2024, 06:20:00 PM »
Egor,
I'm encountering an issue in 2.2.0 when trying to build point clouds - "Internal error in encoding BuildDepthMaps subtask"

Any thoughts?

Tas

  • Newbie
  • *
  • Posts: 48
    • View Profile
Re: Processing Time - Build Point Cloud
« Reply #9 on: November 01, 2024, 06:25:05 PM »
Egor,
I'm encountering an issue in 2.2.0 when trying to build point clouds - "Internal error in encoding BuildDepthMaps subtask"

Any thoughts?

Here is the relevant log file.

Tas

  • Newbie
  • *
  • Posts: 48
    • View Profile
Re: Processing Time - Build Point Cloud
« Reply #10 on: November 01, 2024, 06:48:57 PM »
Update:  "Internal error in encoding BuildDepthMaps subtask" seems to be a bug that occurs when the "save project after each step" box is checked in the Build Point Cloud dialog. 

vineg

  • Newbie
  • *
  • Posts: 29
    • View Profile
Re: Processing Time - Build Point Cloud
« Reply #11 on: November 01, 2024, 07:12:00 PM »
For an alignment preview you could build point cloud only from images (disable laser scans), they can be displayed together. You may even merge them later by selecting in the workspace, right click and Merge... without using Build Point Cloud if everything is OK.

Regards, Egor
« Last Edit: November 01, 2024, 07:14:23 PM by vineg »

Tas

  • Newbie
  • *
  • Posts: 48
    • View Profile
Re: Processing Time - Build Point Cloud
« Reply #12 on: November 01, 2024, 07:18:36 PM »
For an alignment preview you could build point cloud only from images (disable laser scans), they can be displayed together. You may even merge them later without using Build Point Cloud if everything is OK...

Regards, Egor

In 2.1 laser scans depth maps were not calculated, so it is required to rebuild depth maps.

Egor, this might be a very important piece of information for me - thank you. I guess I don't fully understand how the Build Point Cloud process differs when laser scans are included.

I'd really like to better understand the theory behind how laser scans are treated for alignment, depth map generation, and point cloud generation so that 1) I can better troubleshoot misalignments and 2) I can have more confidence in outputs that use survey-grade laser scans that were registered on a control network.

Can you provide any additional information or point me toward a relevant resource? When I explain how Metashape is running laser scan + photo alignments, some people in the industry are quite uncomfortable with the idea that the absolute position of the (fixed-group) laser scans is shifting in space (as a workaround, I'm currently running a very rough script to reposition all cameras and point clouds after processing). Furthermore, finding major misalignments in very high-confidence points makes me question outputs unless each surface is manually checked for alignment.

As it stands, it's very time-consuming to repeatedly process datasets and inspect horizontal/vertical slices of the point cloud. Adding markers in the laser scan image files can be rather challenging when trying to fix misalignments (particularly at greater distances/elevations), since laser scan image resolution is typically quite low.

Bzuco

  • Full Member
  • ***
  • Posts: 225
    • View Profile
Re: Processing Time - Build Point Cloud
« Reply #13 on: November 01, 2024, 07:19:15 PM »
@Tas, that number is just sum of extracted points info from each line in the end of log.
You don't have much options, Ultra High is original photo resolution, so Metashape is trying to make one point from each pixel in photo.
High is half of the photo resolution, Medium is quarter resolution, ...

About that ~559 seconds to load ~5121 MB of depth maps ...preload data may not only mean load from disk, but also extract from archive in which Metashape is storing it and maybe also a little bit of some processing.

Tas

  • Newbie
  • *
  • Posts: 48
    • View Profile
Re: Processing Time - Build Point Cloud
« Reply #14 on: November 02, 2024, 01:16:16 AM »
Thanks again for the responses.

It looks like I might have to restart this project using another software. Despite my best efforts, I cannot reconcile point cloud inaccuracies that are generated in both v2.1.x and v2.2.0. Alignment and tie points look good, but I'm still getting entire walls that are offset by 1/4 - 1/2".

I'm not sure that I understand why the geometry isn't fitted to the laser scan.