Forum

Author Topic: Warning: CL_DEVICE_GLOBAL_FREE_MEMORY_AMD returned too big free memory!  (Read 5410 times)

Mak11

  • Sr. Member
  • ****
  • Posts: 374
    • View Profile
windows 10 , Metashape 1.5.4 with Radeon VII latest 19.8.1 drivers:

[GPU 1] Using device: AMD Radeon VII (gfx906), 60 compute units2019-08-20 20:12:28 Warning: CL_DEVICE_GLOBAL_FREE_MEMORY_AMD returned too big free memory! Data[8]=a8,54,ff,0,0,0,0,0 free_mem_size=16733352 global_mem_size=16580608
, free memory: 12953/16192 MB, OpenCL 2.0

Warning: CL_DEVICE_GLOBAL_FREE_MEMORY_AMD returned too big free memory! Data[8]=54,99,fe,0,0,0,0,0 free_mem_size=16685396 global_mem_size=16580608

Also happens during Refine Mesh process. Warning doesn't happen from the get go but after a job has been started and finished or canceled and a new one is started. Closing & restarting Metashape "fixes" it until it reappears.

Mak


« Last Edit: August 22, 2019, 11:35:37 AM by Mak11 »

Mak11

  • Sr. Member
  • ****
  • Posts: 374
    • View Profile
Re: Warning: CL_DEVICE_GLOBAL_FREE_MEMORY_AMD returned too big free memory!
« Reply #1 on: August 21, 2019, 08:35:12 PM »
Log file

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14813
    • View Profile
Re: Warning: CL_DEVICE_GLOBAL_FREE_MEMORY_AMD returned too big free memory!
« Reply #2 on: August 23, 2019, 01:18:16 PM »
Hello Mak,

Are you using the latest AMD driver version?

As I can see from the log, the processing is not affected, GPU is still utilized. The reason of the warning message is that driver returns the amount of free memory that exceeds total VRAM available.
Best regards,
Alexey Pasumansky,
Agisoft LLC

Mak11

  • Sr. Member
  • ****
  • Posts: 374
    • View Profile
Re: Warning: CL_DEVICE_GLOBAL_FREE_MEMORY_AMD returned too big free memory!
« Reply #3 on: August 23, 2019, 01:54:26 PM »
Alexey,
Yes latest drivers 19.8.1.
There's indeed no performance degradation (I did get a crazy bug while refining a model though but it may not be related to this).
But the warning is definitely inconsistent and never pops up when the app is freshly started and used.

Mak

Mak11

  • Sr. Member
  • ****
  • Posts: 374
    • View Profile
Re: Warning: CL_DEVICE_GLOBAL_FREE_MEMORY_AMD returned too big free memory!
« Reply #4 on: August 23, 2019, 02:57:50 PM »
It may or may not be related:

Before Refine Mesh (Medium 5 iterations):



After:





Closed & restarted Metashape, ran refine mesh on the original model again (same settings etc) and it worked as it should..

Mak

 

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14813
    • View Profile
Re: Warning: CL_DEVICE_GLOBAL_FREE_MEMORY_AMD returned too big free memory!
« Reply #5 on: August 23, 2019, 07:36:51 PM »
Hello Mak,

If you are getting good and bad (completely disastrous) results on the same dataset using the same parameters, it looks more related to the hardware/drivers, moreove, if just the application re-start fixes the problem.
However, if there's a project (alignment and mesh) and source images that can be used to reproduce the problem on AMD Radeon VII, please share it to support e-mail, and we will try to reproduce the problem and investigate the issue.
Best regards,
Alexey Pasumansky,
Agisoft LLC

Mak11

  • Sr. Member
  • ****
  • Posts: 374
    • View Profile
Re: Warning: CL_DEVICE_GLOBAL_FREE_MEMORY_AMD returned too big free memory!
« Reply #6 on: August 23, 2019, 07:48:40 PM »
Alexey,

AMD apparently re-released the 19.8.1 driver on August 20th this time WHQL certified unlike the August 12th version.
Updated them and so far haven't seen the memory warning pop up.
But the Refine Mesh step is still causing the same issue on this particular project. Can't figure out how to reproduce it. Sometimes it works some times it doesn't. Nothing abnormal shows up in the log at all.

Mak

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14813
    • View Profile
Re: Warning: CL_DEVICE_GLOBAL_FREE_MEMORY_AMD returned too big free memory!
« Reply #7 on: August 23, 2019, 07:53:39 PM »
Hello Mak,

I'll try to run multiple refine mesh runs on some random project, just fo case we would be able to catch a similar behavior.
Best regards,
Alexey Pasumansky,
Agisoft LLC

Mak11

  • Sr. Member
  • ****
  • Posts: 374
    • View Profile
Re: Warning: CL_DEVICE_GLOBAL_FREE_MEMORY_AMD returned too big free memory!
« Reply #8 on: August 23, 2019, 07:57:41 PM »
Alexey,

Thanks. That's what I'm currently doing right now. Running 1.5.4 & only tweak enabled is mesh_trimming_radius set to 30 (but this shouldn't affect mesh refinement process). Refineing on Medium 5 steps, smoothing 0.40.

Mak

Mak11

  • Sr. Member
  • ****
  • Posts: 374
    • View Profile
Re: Warning: CL_DEVICE_GLOBAL_FREE_MEMORY_AMD returned too big free memory!
« Reply #9 on: August 23, 2019, 09:32:30 PM »
Alexey,

Rolled back to the previous drivers (19.7.2) and the same issue is present. So it's either a Metashape 1.5.4  problem or this specific project. Will try to downgrade to 1.5.1 and test it (and also redo the project from scratch in 1.5.4).

Mak

Mak11

  • Sr. Member
  • ****
  • Posts: 374
    • View Profile
Re: Warning: CL_DEVICE_GLOBAL_FREE_MEMORY_AMD returned too big free memory!
« Reply #10 on: September 11, 2019, 02:10:12 PM »
It's been a while but I just started getting the same message again while refine a mesh (in 1.6 pre-release with latest 19.9.1 AMD drivers). Will revert back to the August 20 WHQL 19.8.1 package as it seems like all latest non-WHQL drivers have this issue with the Radeon VII.



Mak


Mak11

  • Sr. Member
  • ****
  • Posts: 374
    • View Profile
Re: Warning: CL_DEVICE_GLOBAL_FREE_MEMORY_AMD returned too big free memory!
« Reply #11 on: October 24, 2019, 06:08:47 PM »
Alexey,

Just wanted to point out that Mesh Refinement is still resulting in messed up meshes . This is in Metashape 1.6.0 9357:



Original mesh was generated with the AMD 19.9.3 drivers (non-WHQL) which had the console throwing the same CL_DEVICE_GLOBAL_FREE_MEMORY_AMD returned too big free memory! error during the mesh generation.

I've now re-installed the August 19.8.1 WHQL drivers and I'm rebuilding the mesh to see if I can refine it afterwards without having it explode 🤔

UPDATE: Even with the WHQL 19.8.1 drivers Metashape is now throwing up the memory warning error. Can you guys check this out on your Radeon VII ?



UPDATE 2: Refine Mesh didn't work



MAK
« Last Edit: October 24, 2019, 08:20:45 PM by Mak11 »

Mak11

  • Sr. Member
  • ****
  • Posts: 374
    • View Profile
Re: Warning: CL_DEVICE_GLOBAL_FREE_MEMORY_AMD returned too big free memory!
« Reply #12 on: October 25, 2019, 03:44:30 PM »
Alexey,

I was finally able to refine the model but only after creating a new project from scratch (importing cameras-aligning-generating mesh from depth maps). Other info is that the failed refining attempts took 2x/3x longer to process the destroyed mesh compared to the successful one.

(obviously still getting the memory warnings)

Cheers

MAK

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14813
    • View Profile
Re: Warning: CL_DEVICE_GLOBAL_FREE_MEMORY_AMD returned too big free memory!
« Reply #13 on: October 25, 2019, 04:41:06 PM »
Hello Mak,

You can ignore memory warnings, they do not affect the processing performance.

As for the corrupted model after refinement, it would be helpful for us to get a reproducer.
Best regards,
Alexey Pasumansky,
Agisoft LLC

Mak11

  • Sr. Member
  • ****
  • Posts: 374
    • View Profile
Re: Warning: CL_DEVICE_GLOBAL_FREE_MEMORY_AMD returned too big free memory!
« Reply #14 on: October 25, 2019, 04:46:55 PM »
Alexey,

From the looks of it (Not really 100% sure yet) it seems that refining is failing if it's done with a version of Metashape that is newer than the on used to create the original project in which the mesh was generated. At least that what it is starting to look like given that it worked when I created a brand new project from scratch.

It would be great if AMD was aware of the warning message. It may not affect performance but it sure is annoying to see it constantly pop-up 🤨

MAK