Forum

Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.


Messages - jinjamu

Pages: [1] 2 3 4
1
General / Re: Poor network processing performance
« on: November 15, 2024, 07:19:45 PM »
Thank you for the suggestions, definitely worth trying (I have a 2.5G switch on the way anyway so that's the easy one).  I will also look into the processor undervolting options at a point, however I am wary of trying to change too many things at one go.

So for now on Alexey's suggestion I am eliminating any network induced variables, and looking at performance differences between various modes of working on a single standalone machine, to try to isolate the different performance related factors.  I am seeing differences in performance when using a shared folder using UNC path (on local physical SSD) vs when using the default local drive mapping (e.g D:), with the latter being significantly faster.  Probably this is due to Windows inefficiencies, and maybe it can be resolved by using a performant NAS running with SSDs? 
I am also seeing significant performance differences when running GUI mode vs a Single Node in networking mode (again on a single, non physically networked computer), and this maybe something from Metashape itself.  I have sent logs of my tests to Alexey and am awaiting his comments.

Good news is that one thing I have managed to fix is related to this which was shared by Alexey
https://www.reddit.com/r/XMG_gg/comments/vlqn6d/psa_rendering_tasks_are_moved_to_ecores_when/   
Seems that Windows was relegating processes to slower cores when the specific process is not in "focus" (e.g. not on a maximised, topmost window) and this was causing all sorts of performance variability which was confusing me.  This issue is specific to later Intel processors.  Simply switching to Best Performance power mode solved this, and now Metashape remains running with full resources even when the window is minimized.

Thanks,
John

2
General / Re: Poor network processing performance
« on: November 12, 2024, 08:17:44 PM »
Alexey,

I tried changing the file paths all to UNC, it does not look like a material difference in the timings unfortunately, but I need to recheck.

I also copied the photos to an identical path on both machines and ran the networking test, so this time no photos were being read across the network, only the project files themselves.  There was an improvement in the matching phase, but not in the alignment phase.  I guess this would be consistent with what we were saying regarding the probable bottleneck on the shared folder (photos are read once at the beginning of the alignment?).

I will send you the log, and you can compare it directly with Test 1 log which you already have

Regards

John


3
General / Re: Poor network processing performance
« on: November 12, 2024, 03:29:29 PM »
Alexey,

Thanks for the info, I was not aware (could be a useful tip to speed up initial processing!)

To be certain of the results I have rerun the standalone tests, for both scenarios (project in memory and project pre-saved on local SSD - so NO network processing at all).  There seems to be about 15% increase of time where the project is running pre-saved on SSD vs. purely in memory, and if this extrapolates to larger projects can be considered significant in my opinion.  Compared to this, when running the projects from the shared storage (i.e. both project file and photos on the Razer shared storage), this is MUCH slower.  This happens even for the Razer which is itself providing the shared storage (to remind you that the shared drive is on an internal Razer SSD)

I am wondering if sharing the files from a NAS may help network processing, replacing the Windows sharing.  Windows disk sharing is quite notorious in its poor execution apparently.  Could this be the bottleneck we are experiencing?

I'm not sure this would explain all the results I shared with you though.....

Thanks

John

4
General / Re: Poor network processing performance
« on: November 12, 2024, 12:20:57 PM »
Hi Alexey,

I have just sent you a detailed email with test results and related logs.

Please let me know your thoughts and any suggestions - I am happy to test further....

Regards

John

5
General / Re: Poor network processing performance
« on: November 08, 2024, 07:00:05 PM »
Hi Alexey,

Where is the log for network processing stored please?  Because my first test for network processing came back with an empty log.

Thanks

John

6
General / Re: Poor network processing performance
« on: November 08, 2024, 11:52:48 AM »
Hi Alexey,

To answer your questions:

The shared storage in on a 4Tb SSD on the Razer, which is the machine running the server and the worker also.  I then mapped a drive letter (G:) on both machines (Razer and XMG) to this shared folder.  The project files and the photos are both on G:

For networking I am connecting both machines wired to a wireless access point's 2 Gb Ethernet LAN ports.

The local processing test on the Razer was different only in that it used the proper SSD drive letter (D:) for the project files and the photos.

I did not have logging on but I will switch on and rerun tests.

You say "Would be also interesting to see the processing time for the similar configuration, but if you leave only one node (in network mode) working at a time."  By this do I understand that you want a test on the Razer alone but running in network mode, and with just one Worker on the Razer?

In the meantime following Bzuco's suggestions I ran a test with 3 workers on each machine, then again with 6  on each.  The test with 3 seemed to be producing more CPU and GPU activity on both machines, as compared to the test with one worker on each machine, unfortunately the timing reported in the Chunk Info box was totally wrong, reporting some 5 hours for the process, when it was definitely a lot less, so there is some error in compiling that info for networking mode processing.  (Which may call into question the timings I reported above I guess! - something to look at please). 

The test with 6 and 6 workers failed.  What seems to have happened is at the end of the Matching task all the workers except 1 went into some waiting mode, waiting for just one worker on the XMG, which while showing some CPU activity (3 or 4%) never progressed to completion after 1 hour so I aborted.  Something else to kindly look at.

Finally, another thing to mention is that no single test resulted in the identical number of photos being aligned, even though I was running totally identical alignment parameters.  Small differences (say 10 photos which aligned in one test did not align in another), but still I thought I would mention.

Thanks

John

7
General / Poor network processing performance
« on: November 07, 2024, 02:25:16 PM »
I am doing some preparations for some fairly large scale underwater photogrammetry, looking at multiple models of around 10,000 to 15,000 12Mpix photos, conceivably substantially more photos depending on upcoming trials, to be processed ideally as one model.  I decided to have a look at using the network processing option to see if this could provide a convenient, scalable approach, using networked laptops since processing may actually be happening on board a vessel out at sea.  I set up 2 laptops, one a XMG running a Ryzen 9 5900HX plus Nvidia 3080, the other a Razer running an I9-14900HX plus Nvidia 4090, both with 64Gb of RAM and SSD storage.  Both quite powerful machines as laptops go. 
It took a while to get metashape network processing to work, most of the effort was spent to get the machines to see each other across the LAN (the Metashape part was quite straightforward).  I used the more powerful Razer machine to act as both the Server and a Worker, and the XMG as just a Worker. 
As test data I used a dataset of around 5000 photos which has been previously used to create a model of an underwater shipwreck.  I ran the alignment using identical processing parameters in the Razer standalone, the XMG standalone and both network processed, and compared the alignment timings.  I got the following:
XMG - Matching time 1hr45m, Alignment time 51m
Razer - Matching time 1hr8m, Alignment time 42m
Networked - Matching time 2hr27m, Alignment time 48m
The only setting I changed was to put the Razer (the more powerful computer) to High priority on the management console, everything else I left as default, reasoning that since it is a more powerful machine, it should be given more work to do(?).  (I could not find much in the way of instructions)
So anyway, I was quite surprised that running BOTH machines in parallel resulted in a substantially LONGER matching time than either of the standalone processes, in fact more than twice as long than the Razer alone.  The Alignment time was roughly equivalent in all three tests.  I am sure there are overheads involved, but I doubt they should affect to that extent.
I am of course very interested to understand why this is, could be I am doing something wrong.  It would be great to hear from anyone on this forum who may have experience in such setups.
Thanks in advance
John

8
General / Black texture
« on: April 05, 2023, 01:22:11 PM »
Hi I am encountering a strange problem when texturing a mesh, the texture is turning out all black.  I have tried generating the mesh using interpolation off and on, same result.  The photos are of an underwater object, and because of the visibility the quality is quite low, however another model of an adjacent object taken on the same dive wiuth exactly the same camera etc turned out just fine.
Thanks
John

9
Bug Reports / Re: Blending Textures not using GPU
« on: June 27, 2022, 09:10:20 PM »
Hi Alexey,

Problem is solved!  I had already tried the DDU approach but that did not work.  What fixed things was to upgrade to the latest AMD drivers!  Seems like some incompatibility between Nvidia and older AMD drivers.

Anyway, GPU is back in action.

Thanks for your support on this, highly appreciated!

John

10
Bug Reports / Re: Blending Textures not using GPU
« on: June 26, 2022, 02:17:18 PM »
Hi Alexey,
Thanks for your suggestions. 
I am not so technical so please be patient, but the problem seems tpo be exactly as you are suggesting.
I downloaded and ran GPU-Z and Vulkan box is NOT ticked for the Nvidia processor
I tried a number of older Nvidia drivers and I tried the very last one also (each timewith a clean install) and at no point did Vulkan come up ticked
I downloaded the specific Vulkan RT drivers from the Vulkan site and installed those - same negative result
I also tried installing with AV disabled (I read somewhere that someone solved it this way) but still nothing
Vulkan drivers are not showing in the list of Windows apps (appwiz.cpl)
I'm attaching the extended log as requested https://drive.google.com/file/d/15Op8gYbbanmdLT6HzRa5-ja6OwGyUxPW/view?usp=sharing

I would really appreciate it if you had any ideas on how to get Vulkan working again!

Thanks and regards

John

11
Bug Reports / Re: Blending Textures not using GPU
« on: June 24, 2022, 02:49:15 PM »
Hi Alexey,

I tried disabling the AMD Radeon GPU, and updating to the last drived.  Problem is still there unfortunately.  I still get:

Using device: NVIDIA GeForce RTX 3080 Laptop GPU, 48 compute units, free memory: 14102/16383 MB, compute capability 8.6
  driver/runtime CUDA: 11070/10010
  max work group size 1024
  max work item sizes [1024, 1024, 64]
  got device properties in 0 sec, free memory in 37.858 sec
No GPUs selected. Proceeding on CPU.

Thanks

John

12
Bug Reports / Re: Incremental image alignment on 1.8.3
« on: June 23, 2022, 05:53:45 PM »
Alexey,
I am getting this:

Matching photos...
Warning: Can't resume matching without keypoints
Estimating camera locations...
Estimating camera locations...

So it seems like keypoints on the previously aligned photos are not being detected (even though they are present).  But strange it is also in 1.7.5.......

John

13
Bug Reports / Re: Incremental image alignment on 1.8.3
« on: June 23, 2022, 05:50:09 PM »
Hi,
Yes, keypoints are all there, and the "keep keypoint" option is always enabled.
It's very wierd.  I loaded the metashape file back in 1.7.5 which I have kept installed, and the outcome is identical.  I cannot remember whether the file was saved from 1.8.3, maybe keypoints are treated differently, I don't know.
I wanted to avoid a full realignment if possible, at least for the time being, as I have over 10k hi res images.
Any suggestions?
John

14
Bug Reports / Re: Blending Textures not using GPU
« on: June 23, 2022, 05:22:20 PM »
Hi Alexey!
Another gentle reminder regarding this thread. 
Thanks and regards
John

15
Bug Reports / Incremental image alignment on 1.8.3
« on: June 23, 2022, 05:19:33 PM »
Hi,
Are there any known issues with incremental image alignment on 1.8.3?
I have recently upgraded and I am taking a model built using 1.7.5, and I am trying to add a new set of images.  However the software is not doing any point detection and simply reporting that a number of images have not ben aligned.
Thanks in advance
John

Pages: [1] 2 3 4