I've had a similar problem generating a dense point cloud with an ATI Radeon 5870 GPU, 12-core (2 x X5650) proceessor, and 32 Gb of RAM on a Mac OS X 10.8.5. I didn't get a memory allocation error, the log simply states
...Reconstructing depth...
[CPU] estimating 1549x2340x160 disparity using 775x1170x32f tiles, offset 0
[GPU] estimating 1626x2512x96 disparity using 542x628x32f tiles, offset 0
GPU processing failed, switching to CPU mode
(I attached a slightly longer excerpt from the log file, but the entire log is too big to attach here.)
I've seen others report similar problems with different GPUs, with or without memory allocation errors, but no clear explanation seems to have emerged, except to test the GPU. I ran the suite of Geek3D GPU tests, and the GPU appears to be performing normally. However, the entire dense point cloud generation averaged only 120 million samples per second, which seems very slow. It's a used Mac Pro with a new installation of OS X 10.8.5, and Apple Care is now installing OS X 10.9 and testing the hardware. I'll report back if they find any problems or if this cures it, but any other suggestions or explanations for this problem would be much appreciated.