Metashape Pro Crashing During Image Alignment

[3dMB Ltd]

It looks as though it could be the Gigabyte boards.
https://www.reddit.com/r/gigabytegaming/comments/18mci6a/im_having_some_serious_issues_with_my_z790_aorus/

The explanation bit from the link "the board's VRM apparently can't provide the CPU with high enough voltage in certain situations. The Vdroop for multithreaded load of 14900K may be too low on this board."

[3dMB Ltd]

Thanks Alexey - I have tried both and both tests work fine.

Interestingly, the Prime95 torture test has the P cores running at 5ghz with individual cores spiking at 5400ghz, during the estimate camera locations all P-cores run at 5700ghz.
Ill email you videos showing telemetry

[Alexey Pasumansky]

Has anyone here tried running AVX2 stress-test from Intel Extreme Tuning Utility on the configuration that frequently crashes in Metashape during CPU-intensive operations? Does it pass?

[3dMB Ltd]

Has anyone here tried running AVX2 stress-test from Intel Extreme Tuning Utility on the configuration that frequently crashes in Metashape during CPU-intensive operations? Does it pass?

That runs fine for me

[3dMB Ltd]

I may have found a workaround - not wanting to jump the gun but I think it could be a setting in the Gigabyte BIOS that is causing the issues.
In the BIOS, under CPU settings:
'Turbo Power Limits' is default on 'AUTO'
Changing from 'AUTO' to 'Intel POR' currently seems to be rectifying the issue.

This has worked for me on a couple of small (700-800 image) datasets, now trying on a 63k image dataset.

[3dMB Ltd]

I may have found a workaround - not wanting to jump the gun but I think it could be a setting in the Gigabyte BIOS that is causing the issues.
In the BIOS, under CPU settings:
'Turbo Power Limits' is default on 'AUTO'
Changing from 'AUTO' to 'Intel POR' currently seems to be rectifying the issue.

This has worked for me on a couple of small (700-800 image) datasets, now trying on a 63k image dataset.

Unfortunately Metashape crashed after 24-28 hours into estimating camera locations on the larger dataset test

[colbyrand]

Hi 3dMB Ltd, sorry to hear you are still having issues. I don't believe the issue is caused by the motherboard. I built two identical PCs at my university, apart from the fact that they have different motherboards, to perform Metashape processing. One PC has a Gigabyte Z790 AORUS ELITE AX, and the other has a MSI Mag Z790 Tomahawk. Both PCs are still crashing during Image Alignment...

It is interesting that geodamkru was able to solve the issue by swapping to an Asus Prime z790-P, and Tumno was able to solve the issue by updating the firmware on their NVME SSD. Neither solution have worked for me.

As of now, I am leaning toward agreeing with Alexey Pasumansky that the source of our issues are the Intel Core i9-14900K and Intel Core i9-13900K CPUs, as everyone in this thread has one of these CPUs in their systems, and many other users have had issues with these chips outside of Metashape.

I am currently traveling for work, and won't be back to my university until March, so I am unable to run the Intel Processor Diagnostic Tool or the Prime95 test on my CPU until I get back. If both of these tests pass, but I still believe my CPU to be faulty, does anyone know how easy it would be to RMA my CPUs with Intel? I am hoping that moving down to a Core i7 may fix the issue, but I would need to be able to get a refund on my i9 chips...

Thanks!

[3dMB Ltd]

I reached out to Puget Systems as they say the 14900k ‘should’ be the best CPU for Metashape and is the in their “Suggested System” for Metashape.
Their reply:
“ We no longer test Metashape in our Labs department (and haven't for a couple of years), but we have customers in the field with systems running the Core i9 14900K and the similar i9 13900K without any reports of issues like this. Your best bet would probably be to reach out to Agisoft to see if they know of any issues or have guidance on how to troubleshoot this sort of error. If you want to experiment on your own, it might be worth trying to disable CPU features like Hyperthreading and Efficiency Cores in the BIOS to see if those resolve the problem - not as an ideal long-term solution, but just to see if they might be related.”

[3dMB Ltd]

I reached out to Puget Systems as they say the 14900k ‘should’ be the best CPU for Metashape and is the in their “Suggested System” for Metashape.
Their reply:
“ We no longer test Metashape in our Labs department (and haven't for a couple of years), but we have customers in the field with systems running the Core i9 14900K and the similar i9 13900K without any reports of issues like this. Your best bet would probably be to reach out to Agisoft to see if they know of any issues or have guidance on how to troubleshoot this sort of error. If you want to experiment on your own, it might be worth trying to disable CPU features like Hyperthreading and Efficiency Cores in the BIOS to see if those resolve the problem - not as an ideal long-term solution, but just to see if they might be related.”

Also from Puget Systems:
“We primarily build our 14900K systems on ASUS boards, including the ProArt Z790-Creator Wifi board and the ROG Strix Z690-G Wifi board.”

[3dMB Ltd]

I have had some more success (not complete) with another BIOS setting modification.
Under "AVX settings" is "AVX Optimum" - as standard mine was on "AUTO", changing to "GIGABYTE Optimized" seems to have allowed me to process datasets up to 10k images however it still crashes on my 63k image dataset

[ilia]

I had crashes in alignment stage with Ryzen 5950X CPU.
It happened always when I've tried to reset some alignments or redo some alignments. Is there any chance that I can find these reports stored on the machine? I didn't click "send".

[Alexey Pasumansky]

Hello ilia,

Please check for .dmp and .extra files in the following locations:
on Windows: C:/Users/<username>/AppData/Roaming/Agisoft/Metashape Pro/Crash Reports/pending/
on Linux: ~/.agisoft/metashape pro/Crash Reports/pending
on macOS: ~/.agisoft/metashape pro/Crash Reports/pending

If you can find there files created recently, please send them as attachment to support@agisoft.com.

[ilia]

The folder is empty
 I hope, next time I will save it properly.

[dim4pap]

I also suffered from several crashes with 13900k with asus z690-f motherboard
Most of the times in small projects (500-800 images) there were crashes or pc freezing or pc reboot. In the second try though there was success in the process.
But in my recent ~4500 image project with 33 laserscans, the result was fail, several times.
My pc is in stock settings and I have never done some kind of overclocking. My  2x32 Corsair Vengeance (5600 Mt/s) were working at 4800 Mt/s, that is the stock value.
After lots of hours and hours of reading about similar crashes, my opinion is that it seems that it's a processor's problem in combination with the motherboard or RAM.
Also, searching in the net (mostly in gamers forums) I found out that changing the RAM speed from 4800 to 5600 is recommended and is not considering exactly as overclocking... (it's something I do not wish to do)
And I finally found a way that works for me...
First of all, the stock clock settings of my chip was at 5.5 GHz.
From the official intel's site https://www.intel.com/content/www/us/en/products/sku/230496/intel-core-i913900k-processor-36m-cache-up-to-5-80-ghz/specifications.html
the Performance-core Max Turbo Frequency is 5.40 GHz (!?!?!)
I downloaded Intel(R) extreme tuning utility and set the Performance Core Ratio to 53x
So the first try for the ~4500 image project with 33 laserscans was successful!!!
The second try was to set the Performance Core Ratio to 54x as seen from Intel's site. FAIL again
So I started experimenting changing a bit the settings in the processor and RAM.
I changed again the Performance Core Ratio to 55x as set when I got the PC and changed the RAM speed using profile XMP1 from BIOS. FAIL again.
Finally, keeping the XMP1 (5600 Mt/s) profile I put the Performance Core Ratio to 54x and worked in every try (so far).
in summary:
- clock speed @5300 & ram @4800: success
- clock speed @5400 & ram4800: fail (!!! 13900k frequency and RAM at stock settings!!!))
- clock speed @5500 & ram5600 xmp1: crash (reboot)
- clock speed @5400 & ram5600 xmp1 ok: success

After I found this way out I found various posts like this:
https://www.tomshardware.com/pc-components/cpus/is-your-intel-core-i9-13900k-crashing-in-games-your-motherboard-bios-settings-may-be-to-blame-other-high-end-intel-cpus-also-affected

My recommendation is to set Performance Core Ratio (via Intel's extreme tuning utility) to one or two steps lower than stock settings. If this is not successful try XMP settings in RAM through BIOS

[3dMB Ltd]

Even though I have moved to ASUS ProART board and all crash issues have been rectified,  I am still going through the RMA process with Gigabyte.
I gave Gigabyte access to one of my datasets and they replied by saying:
"We have tested the CPU Over Temperature Protection setting to 80 °C and the software did not crash. Please assist in confirming the outcomes. Thanks"

My reply:
"setting the CPU Over Temperature Protection down to 80 degrees on a CPU that Intel say has a max temp of 100 degrees, it sounds like you are saying that the board is insufficient to run the CPU at its maximum performance potential? This is a field fix to get it working as opposed to addressing the actual issue."

Gigabyte then deleted their message from the esupport portal....

Ultimately, if you are still having issues, you could try reducing the over temp protection but it is only a field fix - I will certainly never be buying another Gigabyte board again