Metashape constantly crashes during Point Cloud creation (MacOS, eGPU (RX 6800)

[mkuttnernpns]

Hello,
I´ve been using Metashape for a couple of months and unfortunately my machine increasingly crashes during Point Cloud creation (which was definitely not the case at the beginning) - maybe someone has a clue how to fix or circumvent my issues. Coming to the details:

1) Hardware: Intel Mac Mini late 2018 (6‑Core 3,2 GHz 6‑Core Intel Core i7; 64GB Ram) plus eGPU (AMD RX 6800 (16gb) in a  Razer Core X case (Thunderbolt 3)). The machine is solely used for drone image processing, no other software is installed except from MacOs, metashape and Firefox.

2) Data used where the issue is recurring: Mainly multispectral imagery (tifs) gained from a P4M drone. Imagery used as "Multi-camera system".

3) Main Project: Reed belt monitoring -- 2 transects, each transect comprised of approx. 1500 (x5) tif images (approx. 250 x 900m each, flight height approx. 66,5m).

4) Issue: Flying the aformentioned transects every month and processing the gathered data as described in the respective metashape tutorial at quality level "High" (becasue I am in need of possibly high-res DEMs). Everything went very well for the first datasets (April (2x), May (2x)) but then Metashape (or my machine or both? - as I don´t know the exact cause) kept starting to crash during Point cloud creation. At the beginning it helped when saving the project under a new name, restarting the machine and repeating the mentioned step. However, this is not the case any more as my machine constantly crashes during this step of processing. In detail, the processing window of metashape freezes for a short moment (approx- 1-2 secs) and then the fans of my eGPU (and machine?) go wild for a fraction of a second (it sounds similar to a person exhaling) before the machine automatically restarts. After restarting, most of the time the infamous apple crash report appears but (at least for me) no viable information can be gathered (just stating something about a kernel panic, cpu1.. that´s it. Sometimes it comes out longer, sometimes very short).  I also tried the etre-check app but couldn´t observe any general problems with this machine..

5) What I tried: Updating of Metashape and MacOs (which supposedly even made it worse somehow?? (as I already described, when the issues started I had luck sometimes and it successfully completed point cloud creation (pcc) after restarting). When setup pcc in quality level "Medium" everything runs smoothely without any issues. The first processing step of photo alignment also works at all quality levels (also highest / ultra). PCC at level "high" also works when disabling the eGPU (via metashape preferences) but takes ages (10 hours for a single dataset compared to approx 40-50 minutes for pcc using the eGPU). My machine uses all cores (respectively threads) at 100% most of the time (either computing with or without the eGPU), showing an avg. temp. of approx. 60 deg. celsius. The only thing I observed is that when computing with eGPU the machine even gets a little bit warmer (up to 63 deg.) -- May this cause the kernel to panic?
I further split up the data just to watch if kernel also panics with much less data to be processed - sometimes it finishes one part without crashing but then crashes during processing the other one.
What I generally do at the very beginning is to re-project my data from WGS84 to a customized ETRS89 projection applying a local geoid before photo alignment<-- but I assume this doesn´t affect further processing at all?

It is really a pity because the eGPU is exclusively used for this purpose in order to significantly save up processing time and energy consumption of my machine and now it seems +/- worthless.
Any hints are very appreciated...

Thanking you in advance, Michael

[mkuttnernpns]

I have an update on the issue:
As I tried to step back to an earlier Version of Metashape (1.8.5), dense ploint cloud creation completed without any issue..

[Alexey Pasumansky]

Hello Michael,

Can you please specify, what options are enabled in the GPU Preferences tab of Metashape window related to the workflow, when Metashape 2.0 crashes and whether Agisoft Crash Reporter window is shown after the mentioned crash? If crash reporter is shown, please submit the report and add the link to this forum thread to the comments field, otherwise please share Metashape logs related to the crashes (it is necessary to specify the path to the log file to be saved in General Preferences tab -> Write log to file field before starting the processing).

[mkuttnernpns]

Dear Alexey,
thank you very much for your reply!
I already had contact with a member of the the Metashape support team (Elizaveta Spiridonova) where I tried to explain my issue several times, however your colleague could not offer a proper solution for the issue, only stating that most likely my hardware would have caused the malfunction. If you like I can forward you the entire communication protocol via email (because it is too long for pasting in this thread)..

Coming to your request: In short - in Metashape preferences --> GPU preferences only the box of my external GPU "AMD Radeon RX 6800 Compute Engine (60 compute units @ 2475 MHz, 16368 MB)   Open CL" is checked, nothing else. Further, there are no other GPU devices listed in this preferences. I also DID NOT check "Use CPU when performing GPU accelerated processing" following the warning message above.
When my machine crashes I haven´t recieved any Metashape crash report file at all, respectively no such window is popping up after restarting the machine or similar. Your colleague asked me to provide a log file for the case when metashape crashed under the described circumstances (during dense point cloud creation on v 2.X), which I already sent over (but I can send it to you as well if you like). I also had a brief look into the log.txt and it unsurprisingly just stops logging by the time of crashing.

What I do recieve every single time after my machine crashes are Apple crash report files - either being rather short (approx. 30-40 lines) only pointing towards the kernel that panicked or being quite long. I honestly don´t know why I do get those 2 different kinds of crash reports on occassion..

Meanwhile I also did some benchmarks and stress tests with my machine which turned out rather fine (no crashes or similar) and my eGPU has been rated at high percentiles compared to similar setups.
Tenperature also does not seem to be an issue, because successful runs with metashape 1.8.5 caused higher kernel temperatures on my machine without panicking than the runs that crashed using v 2.X (I tried 2.0, and most recently 2.0.3.).
If you have any further suggestions how to test my eGPU and/or my machine or any other hints to solve my issue I would greatly appreciate. I could also create another metashape log file or send you an apple crash report file if you like.
Thank you very much again for your reply on my initial post, best regards, Michael

[Low Altitude]

Was there a solution to this in the end, as this is the same set-up I was planning to use. Would like to confirm it does indeed work!

[mkuttnernpns]

As I´m currently not performing any metashape runs I can´t confirm that the latest program version (2.1.X) would perfom better than the last ones (that would cause constant crashes). I´ll be back in this business when at spring.
As I stated before that under 1.8.X it +/ works ok-ish fo me. Only causing crashes when trying to set the highest specs. I have to live along with it because the issues casued are most likely not coming from my hardware setup  - which performs very fine and stable under various other circumstances... Please try to reach the helpdedsk/technicians of agisoft, they probably have a solution. In my case they only stated I should test other hardware which unfortunately is not to my disposal after already having spent several thousand euros ony my current hardware setup.
Sorry that I can´t provide you with more information.
best, michael

[wgreenePIMS]

Any further feedback on this would be great - I am working with a very similar setup (Mac Mini 2018, i7, 32GB RAM, 8GB AMD RX5700 eGPU in a Sonnet Breakaway Puck) and have been experiencing similar crashes that have dramatically intensified recently. I'm running the very latest Metashape version (2.1.0) and I've reinstalled a few times - this does not solve the issue. My crashes occur during depth map building - tie point building and alignment is stable and works great even for very large projects (3-7k images), but I've been unable to generate models even at medium depth map quality for much smaller projects of 1k photos or so. I've had zero other issues with the computer and it seems like the crash problem is tightly related to stability issues during depth map generation.

[mkuttnernpns]

Hi, unfortunately I can´t provide you with further infos on my workflow and projects yet - I´ve already explained my (bad) experiences in the previous posts. I´m sorry for you that you are seemingly facing similar issues (-> also your pocessing history sounds familiar to me when stating that crashes during dense cloud creation notably increased over time). IMHO this points in the direction that not our hardware setup seems to cause the error but maybe changes in the software architecture causing these "instabilities", be it later mac OS versions or metashape versions or both... The only thing I´d advise to you is stepping back to metashape version 1.8.X and try it -- maybe it works out in a more stable way. I know that this is not a solution for the long term but still hoping that my issues will hopefully disappear in the future along with updated versions of mac OS and metashape..
Good luck, best Michael

[wgreenePIMS]

One update on my end - I was able to stabilize depth map generation on the eGPU by using the BuildDepthMaps/pm_enable tweak (set to false). This reverts to using the old depth map generation method (AFAIK) and seems to keep the program from crashing. For now, it's the solution I'm going with because allows me to continue using 2.1. Alexey - can you provide any more information on this? Thank you!

[mkuttnernpns]

@wgreenePIMS -- Wow, tweaking this setting (BuildDepthMaps/pm_enable --> FALSE) solved my issue as well... I can confirm that I´ve successfully processed one of my datasets set to "Ultra High" quality which I´ve chosen for alignment and rather more important for dense point cloud creation steps under Metashape 2.1.

THANK YOU SO SO MUCH for this hint - it safed my day!

This setting is definitely what has affected my hardware and evoked kernel panics all the time.. Further, it makes very much sense that I haven´t experienced those kind of crashes when I started using metashape almost 2 years ago... And, from my side I´ve always ruled out that any faulty hardware, being it my eGPU or my machine would cause those issues because it has worked at the beginning (as I already stated several times in my prrevious posts) and I constantly performed stress tests on my system over time.
Again thank you very much for this ultra-valuable post -- this is a virtual life safer for processing my data! May I ask how did you figure out to tweak this specific parameter?

So glad this issue is at least identified by now and there is a proper workaround!

Best, Michael

[wgreenePIMS]

Hi Michael,

Glad the tweak worked for you as well. That particular tweak has been around for a while (I guess since the new depth maps meshing technique came out around version 1.8 ) and I experienced issues with the new depth maps when they were first implemented, when they tended to cause strange folds and double-surfaces in some meshes, so I had to use that tweak to keep our production workflow functioning well. Those problems seem to have been fixed, and the new depth maps method is definitely an improvement in detail when it doesn't cause crashes, so I've switched the tweak off on most of our machines. The Mac/eGPU combination crash issue and your fix of going back to 1.8 made me wonder if the new depth map method was the culprit, which it clearly is based on our experiences. While this workaround is okay for now, I'd really like to see this issue addressed by Agisoft.

Will

[Forester123]

Hello,

I also use a Macbook Pro with an egpu and recently I can no longer generate a point cloud. Photos can be aligned and error reduction carried out, but in the middle of the point cloud creation process the MacBook shuts down.
I haven't worked on any projects in the last few weeks, but before that I never had a problem (up to Version 2.1).

It's not clear to me where I can find the point builddepthmaps/pm_enable? Could someone please explain to me or show me with a screenshot where I can make the setting?

Thank you.

Tobias

[mkuttnernpns]

Hi,
you can find it in "Metashape Preferences" / "Advanced" right at the bottom --> "Tweaks".
See also metashape manual - pages 220, 221.
Good Luck and best regards, Michael