Forum

Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.


Messages - andyroo

Pages: [1] 2 3 ... 30
1
Hi Agisoft folks!

VERY glad to see copc format supported. My feedback so far:

1) metashape copc export is the only way I can preserve confidence and normals and get my point clouds from metashape into .copc format (PDAL mangles/deletes user bytes) and lascopcindex64 throws coordinate system errors.

2) exporting COPC is about 40x slower than outputting normal laz files, which makes exporting large dense clouds.... problematic. (minutes turn into hours, and hours into days)

3) exhorting COPC with Metashape is still about  3-4x faster than converting .laz exported by metashape to copc with lascopcindex64, which also preserves normals/confidence but makes the data unusable with any other data because it destroys the CRS

4) exporting COPC with Metashape is about 60x slower than converting .laz files with PDAL/QGIS, which does a great job with RGB, XYZ, and classification AND shrinks filesize by 25% but I'm pretty sure the only reason the files are smaller is because it drops the userbytes due to deficiencies in PDAL support of data stored in user bytes.

2
Feature Requests / Re: Instant NERF?
« on: March 14, 2024, 12:58:42 AM »
This (Photogrammetry+NeRF fusion) is something I've been thinking about for a while - with my geologist's inadequate understanding of the mathematical specifics of both of these machine-vision algorithm families.

Specifically how NeRFs could inform MVS, or more generally the  photogrammetric dense reconstruction algorithm(s). From what I understand NeRFs, from Plenoxels to 3D Gaussian Splatting, still use the/a SfM-derived sparse cloud and generate dense clouds using trained models with much more poorly constrained geometry than MVS-type algorithms... BUT - it seems like coregistration would be relatively trivial if the sparse points are shared, and if the MVS dense points with highest confidence and lowest error are used to further constrain the NeRF such that the NeRF can contribute valuable information where MVS falls apart - like with specular reflections or low contrast surfaces (roads/water/sand from my aerial photogrammetry perspective).

I feel like the NeRF and photogrammetry families are walking separate paths with a fence in-between and they need to make a love child.

3
When I draw a polygon in QGIS, and export as a geopackage (or shapefile) in EPSG:6318, then try to import into Metashape while setting outerBoundary as the boundary type it imports successfully but does not set the outer boundary.

I can set the outer boundary after import manually or with a python script.

I'm pretty sure I tried it as a shapefile too and tried a manual import. In all cases I had to set the boundary after import.

sample gpkg and jpg attached.

jpg shows logfile with "boundary_type = OuterBoundary" and context menu with /Set Boundary Type/None checked (project state immediately after import)

4
Feature Requests / Re: Add "Build Tie Points" to batch processing
« on: January 19, 2024, 10:23:23 PM »
+1 - I also requested that here in July and got a +1 by ManyPixels - so that's at least three of us hoping for the feature.

5
Hi Alexey,

I replied on freshdesk with basically the same text but posting here for other folks in case it's useful.

TLDR; I updated to 2.1.0.17532 and launched host/monitor/worker with admin privileges :o and was able to finish the align.finalize step

Three of the nodes the host rotated through had enough RAM (one with 192GB and two with 256GB), and disk swap/virtual memory was enabled (175GB to NVMe) on the one with 64GB. I watched the node with 64GB for a while yesterday before it crashed and the RAM got up to just about full then it looked like disk swap kicked in.

I didn't see crash reports on any of the nodes, and it looked like the host was hanging up on them based on the messages in the host log:

recv: An established connection was aborted by the software in your host machine (10053)
failed #0 AlignCameras.finalize (1/1): Connection closed
send: An existing connection was forcibly closed by the remote host (10054)


After I updated the server and a worker node to 2.1.0.17532, and edited the JSON to enter that version, AND launched the host and worker and monitor processes with admin privileges THEN the job finally completed, although it looks like the worker crashed and generated an error report before recovering and finally finishing.

6
Hi jrp,

I only have one comparison, using Metashape 1.5.5 on a 3960x ThreadRipper with 256GB RAM and 2x RTX 2080 Super GPUs. I benched at 2400MHz and 3000MHz with the Puget Systems extended Metashape benchmark. I've been modding the script to work with later versions but I haven't run a bench at multiple RAM speeds on any other configuration.

Using the extended benchmark, I had these total processing times:

Project/Processing time                          2.4GHz     3GHz      Difference

Park Map
Total Processing Time:                           10974.9     9111.7     20.4% faster

School Model
Total Processing Time:                            2975.1      2584.1     15.1% faster

School Model w/ DepthMaps
Total Processing Time:                            8640.8      7631.8      13.2% faster



7
General / Re: Metashape Network Processing
« on: January 12, 2024, 12:26:28 AM »
That's interesting - so your node is frozen until you hit the spacebar... That sounds very similar to a problem I just reported here where my nodes are freezing and being disconnected, but I just found that I was able to "unfreeze" a node by hitting <enter> in the console window on the node.

Nodes and host are all windows 10 or Windows 11, Metashape Pro 2.1.0.17526, files are on a NVMe RAID on the host, connected to nodes via 10Gbe

[edited] ..note - when I updated to 2.1.0.17532 and launced workers and host in admin mode I was able to complete the align.finalize step

8
Hi Agisoft folks. I'm running version 2.1.0.17526 on host and workers, on Windows 10 (host and some workers) and Windows 11 (a few workers).

TLDR; I'm having problems during align.finalize with freezing/disconnecting workers, and after an overnight freeze without disconnect, I was able to restart worker progress today by pressing the <enter> key in the console window?!

Additional details below, and I filed a freshdesk ticket (#194391) a couple days ago and have been updating it. Some information is repeated here in case it helps anyone or anyone has any insight...

On my first network align of ~26,000 images everything went fine until the align.finalize step and then all of the workers were periodically disconnected about 275 times with a message something like this on the monitor (on the host machine):

2024-01-06 14:40:04 192.168.88.201:58729] recv: An established connection was aborted by the software in your host machine (10053)
2024-01-06 14:40:04 [192.168.88.201:58729] failed #0 AlignCameras.finalize (1/1): Connection closed
2024-01-06 14:40:04 [192.168.88.201:58729] send: An existing connection was forcibly closed by the remote host (10054)
2024-01-06 14:40:04 [192.168.88.201:58729] worker removed

...

after about 250 times, I disconnected all nodes, saved the batch, updated network card drivers, restarted the host and a single node, and tried again. That was the day before yesterday. Overnight the worker disconnected another 25 or so times, then finally finished the first step(?!)

So that's weird - why would it fail 275 times then suddenly work? But it gets weirder -  I noticed yesterday around 5pm that the worker node appeared frozen - there was no activity on the host/monitor on the worker/details progress graph, and the last update was at 2024-01-10 14:59:30. I left it alone until a few moments ago, and was surprised to see that the worker node never disconnected. I noticed that the host showed this:
2024-01-10 14:59:30 adjusting: !xx

while the worker showed this:
2024-01-10 14:59:30 adjusting: !x

Out of frustration, or desperation, or I don't know why, I hit <enter> in the console (cmd) window of the worker, and then it showed this!
2024-01-10 14:59:30 adjusting: !xxx

AND weirdest of all (to me) the host graph started updating again, and the worker appears to be running normally...

The current worker log looks like this (with around 25 more disconnects above that I didn't paste in)

...
x2024-01-10 08:18:51 Error: Aborted by user
2024-01-10 08:18:51 processing failed in 1950.93 sec
disconnected from server
connected to 192.168.88.205:5840
registration accepted
2024-01-10 08:19:48 AlignCameras.finalize (1/1): subtask = finalize, adaptive_fitting = off, level = 6, cache_path = //SFM-HOST/Network_SfM/psx/SBC_master_all_images_guided_5k.files/0/align.1
2024-01-10 08:20:26 3 blocks: 20308 146 2
2024-01-10 08:34:38 block: 14 sensors, 20456 cameras, 381612284 points, 1593718574 projections
2024-01-10 08:34:38 block_sensors: 0.0118561 MB (0.0127029 MB allocated)
2024-01-10 08:34:38 block_cameras: 7.95941 MB (11.8527 MB allocated)
2024-01-10 08:34:38 block_points: 17468.8 MB (21044.8 MB allocated)
2024-01-10 08:34:38 block_tracks: 1455.74 MB (1455.74 MB allocated)
2024-01-10 08:34:38 block_obs: 72954.6 MB (72954.6 MB allocated)
2024-01-10 08:34:38 block_ofs: 2911.47 MB (2911.47 MB allocated)
2024-01-10 08:34:38 block_fre: 0 MB (0 MB allocated)
2024-01-10 08:37:58 adding 353608275 points, 0 far, 1612701 inaccurate, 14316 invisible, 179 weak
2024-01-10 08:40:59 adjusting: !x[192.168.88.205:5840] recv: An existing connection was forcibly closed by the remote host (10054)
x2024-01-10 08:52:17 Error: Aborted by user
2024-01-10 08:52:17 processing failed in 1949.04 sec
disconnected from server
connected to 192.168.88.205:5840
registration accepted
2024-01-10 08:53:15 AlignCameras.finalize (1/1): subtask = finalize, adaptive_fitting = off, level = 6, cache_path = //SFM-HOST/Network_SfM/psx/SBC_master_all_images_guided_5k.files/0/align.1
2024-01-10 08:53:53 3 blocks: 20308 146 2
2024-01-10 09:08:08 block: 14 sensors, 20456 cameras, 381612284 points, 1593718574 projections
2024-01-10 09:08:08 block_sensors: 0.0118561 MB (0.0127029 MB allocated)
2024-01-10 09:08:08 block_cameras: 7.95941 MB (11.8527 MB allocated)
2024-01-10 09:08:08 block_points: 17468.8 MB (21044.8 MB allocated)
2024-01-10 09:08:08 block_tracks: 1455.74 MB (1455.74 MB allocated)
2024-01-10 09:08:08 block_obs: 72954.6 MB (72954.6 MB allocated)
2024-01-10 09:08:08 block_ofs: 2911.47 MB (2911.47 MB allocated)
2024-01-10 09:08:08 block_fre: 0 MB (0 MB allocated)
2024-01-10 09:11:28 adding 353608275 points, 0 far, 1612701 inaccurate, 14316 invisible, 179 weak
2024-01-10 09:14:34 adjusting: !xxxxxxxxxxxxx!x!x!x 0.823228 -> 0.75973
2024-01-10 10:40:05 disabled 1 points
2024-01-10 10:43:18 adding 1641944 points, 117372 far, 1628257 inaccurate, 14315 invisible, 180 weak
2024-01-10 10:43:18 optimized in 5510.16 seconds
2024-01-10 10:43:18 f 8429.71, cx 27.8211, cy 29.5716, k1 -0.118365, k2 0.125958, k3 0.0315367
2024-01-10 10:43:18 f 8430.09, cx 25.449, cy 29.4083, k1 -0.118596, k2 0.127053, k3 0.030009
2024-01-10 10:43:18 f 8428.33, cx -0.407983, cy 16.2905, k1 -0.11752, k2 0.1239, k3 0.0364546
2024-01-10 10:43:18 f 8433.91, cx -2.50267, cy 15.1906, k1 -0.117851, k2 0.125403, k3 0.0334202
2024-01-10 10:43:18 f 8426.79, cx 6.52533, cy 14.7927, k1 -0.118174, k2 0.12028, k3 0.0464695
2024-01-10 10:43:18 f 8427.1, cx 10.2056, cy 13.7934, k1 -0.11839, k2 0.123723, k3 0.0398974
2024-01-10 10:43:18 f 8427.91, cx 39.1095, cy 31.2033, k1 -0.118537, k2 0.128099, k3 0.0267767
2024-01-10 10:43:18 f 8429.72, cx 20.7913, cy 33.5567, k1 -0.117826, k2 0.123211, k3 0.0376948
2024-01-10 10:43:18 f 7366.84, cx -13.8719, cy 18.8519, k1 -0.0897347, k2 0.117177, k3 -0.0388326
2024-01-10 10:43:18 f 8428.96, cx 22.1275, cy 34.2011, k1 -0.11777, k2 0.12464, k3 0.0328493
2024-01-10 10:43:18 f 8748.63, cx 4.73646, cy -35.1598, k1 -0.114165, k2 0.15207, k3 0.0610519
2024-01-10 10:43:18 f 8193.61, cx 0, cy 0, k1 0, k2 0, k3 0
2024-01-10 10:43:18 f 10242, cx 0, cy 0, k1 0, k2 0, k3 0
2024-01-10 10:43:18 f 8193.61, cx 0, cy 0, k1 0, k2 0, k3 0
2024-01-10 10:46:12 adjusting: !xxxxxxxxxxxxx!x!x!x 0.74288 -> 0.742355
2024-01-10 12:13:53 final block size: 20456
2024-01-10 12:17:13 adding 353617053 points, 0 far, 1613259 inaccurate, 14284 invisible, 179 weak
2024-01-10 12:17:13 (3 px, 2 3d) sigma filtering...
2024-01-10 12:20:19 adjusting: !xxxxxxxxxxxxx!x!x!x 0.822517 -> 0.759586
2024-01-10 13:47:34 point variance: 0.841952 px, threshold: 2.52586 px
2024-01-10 13:50:44 adding 1449532 points, 9624987 far (2.52586 px threshold), 1323685 inaccurate, 2899 invisible, 91 weak
2024-01-10 13:51:13 removed 4 cameras: 20395, 20397, 20398, 20399
2024-01-10 13:51:13 removed 4 stations
2024-01-10 13:53:46 adjusting: !xxxxxxxxxxxxx!x 0.278704 -> 0.27798
2024-01-10 14:54:00 point variance: 0.306791 px, threshold: 0.920373 px
2024-01-10 14:57:03 adding 1254004 points, 14764293 far (0.920373 px threshold), 1111041 inaccurate, 1055 invisible, 137 weak
2024-01-10 14:59:30 adjusting: !xxxxxxxx


[edit]
Update - the host killed the process when it was about 87% complete. Looks like I might have to restart from scratch and process on a local machine. Going to try to copy everything to the worker node and run host/worker/monitor all on one machine so I don't have to restart everything but I need to modify paths... not sure how that's going to work...


9
remote pc didn't crash it continued processing. that's what the "oblivious node" screenshot showed. Unfortunately the host pc crashed. I'm rebuilding that right now - looks like corrupted windoze install maybe

10
Hi Metashape folks!

I'm running an alignment and I had a node doing AlignCameras.update and it got disconnected from the server (three other nodes show but not that one) - another node took over the operation, but the disconnected node is still going through the motions. is this unusual? what can I do to keep it from happening? Is there anyway to reconnect the disconnected node since it's still chugging away? I noticed that other nodes showed disconnect messages too but are still there...

Attached screenshots of the monitor showing the disconnection and the disconnected node still processing 22 minutes later...

11
Howdy Agisoft folks!

I'm digging into the guts of network processing on a 4-node cluster I built and I see that in the AlignCameras.update portion, I'm limited to a single node, and when I dive into that node, it's working on a single core, but using 80% of system RAM (256GB). Wondering if I would be able to process this faster on a node with faster RAM and CPU frequencies, but that node also has less RAM (64GB). The AlignCameras.update step appears to be a significant bottleneck for this workflow (processing multiple overlapping PPK flights together - about 26,000 images).

Thanks for making awesome software! :-)

12
Hi Metashape folks. I recently configured a (windoze) network with 4 nodes and I'm wondering if I need to manually set the GPU mask at 3 to use both GPUs or if the "default GPU mask" uses the number of detected GPUs (and if there's a way to see what the default is for a given node).

Thanks!

13
I have a chunk with 78218 cameras, 7372 of which are disabled and have no estimated positions. When I export all values (all checkboxes) from the GUI (12 decimals of precision) in tab-delimited format, all of the "enabled" flags are set to 1 regardless of whether the image is enabled or disabled.

-EDIT- when I calculate this flag as 1 or 0 based on whether the "estimated lat" value is present or absent, regardless of the value all images still import as enabled.

14
gradual selection/filter by confidence on the dense cloud might do what you want - you can get an idea if that would work for you by looking at the dense cloud confidence rather than RGB values. You can either delete those values or reclassify them and omit them from any DEM derivatives.

15
Can anyone explain for the chunk.matchPhotos method:
  • what are the workitem_size_cameras, workitem_size_pairs, and max_workgroup_size attributes?
  • how do they affect matching results?
  • (if applicable) what are the limits/guidelines for these values based on image mpx and GPU RAM?

I'm wondering how best I can use this method + chunk.triangulateTiePoints for "densifying" tie points after initial alignment and before optimization to increase intra-camera-group matches(before generating dense cloud).

For example, I have chunks with 10s of camera groups with thousands of images each that are coaligned (incrementally) using generic preselection 60k keypoints limit, 0 tiepoints limit.

Experimenting with a single camera group after optimization, by copying the chunk and deleting the other groups, I was able to increase the number of intra-group tiepoints by running chunk.matchPhotos then chunk.triangulateTiePoints with either the same matching parameters as before, or by switching to guided matching, for example:

Code: [Select]
chunk.matchPhotos(downscale = 1, generic_preselection=True, reference_preselection=False, keypoint_limit_per_mpx=4000, tiepoint_limit=0, guided_matching=True, reset_matches = True)
chunk.triangulateTiePoints()

Then if I re-filter with the same gradual selection parameters that I used before optimization I end up with anywhere from 2x to 10x the original number of tie points without changing intrinsics or extrinsics. This is nice because some of my early image collections didn't previously have enough projections to reconstruct well (many missing images from dense cloud), and using this method dramatically increases the projections on many of the trouble images without a huge increase in processing time. But I'm wondering if there are downsides that I'm not realizing to doing this, and if there are even more useful ways for me to take advantage of this by playing with the  workitem_size_cameras, workitem_size_pairs, and max_workgroup_size elements.

I'm also wondering if I can use this technique to increase inter-group matches by combining specific groups of cameras and passing that list to the 'cameras' variable of the matchPhotos method, and what the difference is between reset_matches = True vs False. "True" seems to re-run the entire matching process on every image but generated 360k matches vs 24k). Is that because there was a tiepoint threshold that was limiting new tiepoints (many of the original tie points were with images from other groups)?

If anyone has some deeper knowledge of the inner workings of this method, I'd be most grateful for your insight.

Pages: [1] 2 3 ... 30