Forum

Author Topic: PS 1.03 Build 1832 CL_MEM_OBJECT_ALLOCATION_FAILURE  (Read 10517 times)

andyroo

  • Sr. Member
  • ****
  • Posts: 448
    • View Profile
PS 1.03 Build 1832 CL_MEM_OBJECT_ALLOCATION_FAILURE
« on: March 14, 2014, 08:59:38 PM »
I just updated from 1.0.2 build 1824 to 1.0.3 build 1832 and in my process log I am seeing a lot of this error, which is not in logs prior to the update:

ocl_engine.cpp line 298: clEnqueueNDRangeKernel failed, CL_MEM_OBJECT_ALLOCATION_FAILURE

Here's what the start of my log for the latest run reads like:

Code: [Select]
Using device: GeForce GTX 560 Ti, 8 compute units, 1023 MB global memory
  max work group size 1024
  max work item sizes [1024, 1024, 64]
  max mem alloc size 255 MB
Using device: Tahiti, 32 compute units, 2048 MB global memory
  max work group size 256
  max work item sizes [256, 256, 256]
  max mem alloc size 512 MB
initializing...
skipping pair: camera visibility test
skipping pair: unable to estimate disparity range
skipping pair: unable to estimate disparity range
skipping pair: unable to estimate disparity range
skipping pair: unable to estimate disparity range
skipping pair: unable to estimate disparity range
skipping pair: unable to estimate disparity range
skipping pair: unable to estimate disparity range
skipping pair: unable to estimate disparity range
skipping pair: unable to estimate disparity range
skipping pair: unable to estimate disparity range
skipping pair: unable to estimate disparity range
skipping pair: unable to estimate disparity range
skipping pair: unable to estimate disparity range
skipping pair: unable to estimate disparity range
skipping pair: unable to estimate disparity range
skipping pair: unable to estimate disparity range
skipping pair: unable to estimate disparity range
skipping pair: unable to estimate disparity range
skipping pair: unable to estimate disparity range
skipping pair: unable to estimate disparity range
skipping pair: camera visibility test
selected 273 cameras from 298 in 13.569 sec
Loading photos...
Reconstructing depth...
[GPU] estimating 1770x3160x96 disparity using 590x632x8u tiles, offset 0
ocl_engine.cpp line 298: clEnqueueNDRangeKernel failed, CL_MEM_OBJECT_ALLOCATION_FAILURE
GPU processing failed, switching to CPU mode
[CPU] estimating 1770x3160x96 disparity using 885x1054x8u tiles, offset 0
[GPU] estimating 2618x3727x96 disparity using 873x746x8u tiles, offset 0
[CPU] estimating 2976x4256x128 disparity using 992x1064x8u tiles, offset 0
timings: rectify: 1.104 disparity: 14.788 borders: 0.103 filter: 0.357 fill: 0
[GPU] estimating 3749x3144x192 disparity using 625x524x8u tiles, offset 0
ocl_engine.cpp line 298: clEnqueueNDRangeKernel failed, CL_MEM_OBJECT_ALLOCATION_FAILURE
GPU processing failed, switching to CPU mode

Running 64 bit version on Windows 7x64 w 128GB RAM, Nvidia GeForce GTX 560 Ti and AMD Radeon HD 7900. Configuration hasn't changed for months and I am running the same area, same chunks, just new pictures. Driver versions (from device mgr):

AMD 13.152.1.8000
NVidia 9.18.13.1422

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 15138
    • View Profile
Re: PS 1.03 Build 1832 CL_MEM_OBJECT_ALLOCATION_FAILURE
« Reply #1 on: March 15, 2014, 01:18:42 AM »
Hello andyroo,

Could you please try running the same processing with only one GPU enabled in OpenCL tab to check which card is causing the problem?
Best regards,
Alexey Pasumansky,
Agisoft LLC

andyroo

  • Sr. Member
  • ****
  • Posts: 448
    • View Profile
Re: PS 1.03 Build 1832 CL_MEM_OBJECT_ALLOCATION_FAILURE
« Reply #2 on: March 16, 2014, 02:24:38 AM »
Hmm.. I cancelled the job, disabled each one separately and restarted and tried to reproduce the error. No luck. Then I enabled both again and so far the error has not resurfaced. Will keep you posted if the problem occurs again and I am able to narrow down which GPU.

Mfranquelo

  • Full Member
  • ***
  • Posts: 171
    • View Profile
Re: PS 1.03 Build 1832 CL_MEM_OBJECT_ALLOCATION_FAILURE
« Reply #3 on: April 09, 2014, 12:19:46 PM »
Dear Alexey,

Im getting the same error right now, i've just added an scale bar to my model and removed some other markers. Dense point cloud generation takes 7 hours right now, it took like half an hour with this very same model. I didn?t change graphic drivers or whatsoever. 
Im running Agisoft 64 bits
Using 6 CPU cores out of 8, and Two GPU cores

Intel Core i7 3770 CPU 3.40GHZ 3.90GHZ
32GB RAM
64bits


The Log is attached.


Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 15138
    • View Profile
Re: PS 1.03 Build 1832 CL_MEM_OBJECT_ALLOCATION_FAILURE
« Reply #4 on: April 09, 2014, 12:23:47 PM »
Hello Mfranquelo,

Can't see the error message in the log file? Are you sure that the time difference is not caused by the different Quality selected for the dense cloud generation?
Best regards,
Alexey Pasumansky,
Agisoft LLC

Mfranquelo

  • Full Member
  • ***
  • Posts: 171
    • View Profile
Re: PS 1.03 Build 1832 CL_MEM_OBJECT_ALLOCATION_FAILURE
« Reply #5 on: April 09, 2014, 01:00:15 PM »
Im not getting this one exactly: CL_MEM_OBJECT_ALLOCATION_FAILURE - im sorry

Just skipping pair: unable to estimate disparity range which i wasnt aware of before.
Im 100% that i am using "High".

What i did was this:
1. Align high, dense point cloud generation high, generate mesh
2. Realise that the model wasnt scaled
3. Choose 2 markers on mesh
4. Set scale bar
5. Optimize ( I then had to generate dense point cloud all over again)
6. Its taking like 3 hours in comparison to the half an hour that took the first time

----------------------

I re-aligned the points again from the beginning, using scale bar etc and it takes 1/2 hour now, i dont know whats changing  ???