Forum

Author Topic: System hard freeze while GPUs processing with ATI7970 if using computer  (Read 15404 times)

andyroo

  • Sr. Member
  • ****
  • Posts: 438
    • View Profile
Hi everyone,

I am wondering if anyone else has had a problem with system instability using the ATI HD7970 cards.

I have a problem where, if I try to use my workstation for anything while processing a model with PhotoScan, my system will freeze (sometimes in minutes, sometimes hours) if I try to do anything else. I have this problem in 0.9.1 and the 1.0 (1736) beta. Haven't tried going back to 0.8.5. Didn't occur with my NVidia cards. I thought it might be a temp issue and was carefully monitoring temps on this run, and both cards were both pretty low (60s and 70s) when this happened.

I am starting to wonder if it's a defective card, but have no idea how to figure that out. Nothing in the event log. Symptoms are that for a few seconds before the freeze the system stutters (redraw, mouse, sound if playing), then it becomes unresponsive. Sometimes one monitor blanks, but not always the same one - which makes me think maybe not the card.

I have more than enough power for both cards - I was underpowered before and that just shut off my machine.

Any thoughts?

Porly

  • Jr. Member
  • **
  • Posts: 77
    • View Profile
Re: System hard freeze while GPUs processing with ATI7970 if using computer
« Reply #1 on: September 06, 2013, 11:04:42 PM »
Hello andyroo,

I had a similar problem after shipping my computer to brasil. I forgot to dismount the graphic card for the transport. As I ran my pc for the first time, the screen freezed when GPU was used.

The GPU was defect. A screw was solved, and there were no way to fix it.

Try to run other application with a big GPU requirement. If you have the same problem, I would guess that your card is defect.

For now I am using two new 7970's and it works fine...

In which step your pc is hanging? Since GPU is only used for generating Depth Maps and in Model View when VBO support is enabled.

Disabled Crossfire??

Newest Drivers??


Wishgranter

  • Hero Member
  • *****
  • Posts: 1202
    • View Profile
    • Museum of Historic Buildings
Re: System hard freeze while GPUs processing with ATI7970 if using computer
« Reply #2 on: September 06, 2013, 11:38:53 PM »
try run some benchmarks that heavy taxe the GPU for at least half hour like http://unigine.com/products/valley/ if it crash then its somehing with card.

AS Porly mention drivers etc, and COOLING of the card download the MSI aftherburner and monistor GPU temperature..... have your MB integrated video card ??
----------------
www.mhb.sk

andyroo

  • Sr. Member
  • ****
  • Posts: 438
    • View Profile
Re: System hard freeze while GPUs processing with ATI7970 if using computer
« Reply #3 on: September 07, 2013, 12:59:42 AM »
Mobo does not have integrated video.

I was monitoring GPU temps up to the crash this time and they were 60-65 on the primary card and 75-85 on secondary card. sounds well below crash thresholds (although I am trying to cool the secondary card down).

PC hangs when generating dense point cloud. I was watching GPUs pretty closely, they were definitely in use.

Crossfire is not enabled.

Will run benchmarks. have a ticket into Gigabyte too.


Wishgranter

  • Hero Member
  • *****
  • Posts: 1202
    • View Profile
    • Museum of Historic Buildings
Re: System hard freeze while GPUs processing with ATI7970 if using computer
« Reply #4 on: September 07, 2013, 01:35:13 AM »
hmm temps are OK, can go up to 85-90 deg. how big is the dataset (photos + res + ammount and reconst settings ) and how much ram you have ?? CPU temp ? try disable ALL Cpu cores for reconstruction in settings and  try install newest drivers ( even BETA 13.10 ) then use just ONE card. run the dataset, disable 1 enable 2nd card rerun.....
----------------
www.mhb.sk

andyroo

  • Sr. Member
  • ****
  • Posts: 438
    • View Profile
Re: System hard freeze while GPUs processing with ATI7970 if using computer
« Reply #5 on: September 10, 2013, 12:16:27 AM »
latest observation makes me think its likely a defective card: I was in "generating mesh" step of photoscan - CPU and GPU both were at ~15-20% max use, temps were all fine (30-44°C for CPU cores, 60-70°C for GPU cores), and as I was posting my message about speed of "generating mesh" step, my system froze. For the first few seconds, the keyboard lagged, but mouse still worked, then mouse froze, then one monitor went black and lost signal. Other monitor just frozen.

System was rock-solid before swapping out the two NVidia for two ATI HD7970 cards. The weird thing to me is that it has crashed probably close to 10 times on PS, but ONLY when I try to do something else too.

Wishgranter

  • Hero Member
  • *****
  • Posts: 1202
    • View Profile
    • Museum of Historic Buildings
Re: System hard freeze while GPUs processing with ATI7970 if using computer
« Reply #6 on: September 10, 2013, 12:19:46 AM »
what PSU you have ??? and what all devices are inside ?? send me precise PSU model.....
----------------
www.mhb.sk

andyroo

  • Sr. Member
  • ****
  • Posts: 438
    • View Profile
Re: System hard freeze while GPUs processing with ATI7970 if using computer
« Reply #7 on: September 10, 2013, 01:40:07 AM »
what PSU you have ??? and what all devices are inside ?? send me precise PSU model.....

Primary PSU is Dell T7500 NPS-1100BB / 0R622G 1100W PSU
Secondary PSU is FSP Booster X5 450W

Inside:

2 Intel X5647 2.93 GHz Quad Core Processors
12x16GB = 192 GB Wintec server series DDR3 1333MHz REGISTERED ECC RAM
Stock fans plus EVERCOOL SB-F3 Case Cooling Fan
1 Corsair 360GB SATA3 SSD (CSSD-F360GBGS-BK)
4 1TB 7200 RPM Western Digital HDD (WD1002FBYS)
1 DVDRW
1 HighPoint RocketRAID 622 2-port eSATA card.

on eSATA:

Sans Digital TR5M+B 5 bay TOWERRAID with
 5 3TB Seagate Constellation ES 7200RPM SATA3 HDD (ST33000650NS)
 configured as two 3TB mirrored RAIDs with a hot-swap spare.

And for completeness:

2 ViewSonic VA2702w monitors (1 per card)

Matt

  • Full Member
  • ***
  • Posts: 104
    • View Profile
Re: System hard freeze while GPUs processing with ATI7970 if using computer
« Reply #8 on: September 10, 2013, 07:18:33 AM »
I am running a combo of 7970 and gtx590 cards. In the last two weeks i have been having similar issues. In my case I would get a lock up when viewing multimedia (flash etc)on the web while processing on the graphics cards in Pscan. Same symptoms of flickering display ending up with the BSOD. I have manually switched of automatic driver updates and installed the latest beta catalyst drivers. Still had crashes but wiped all drivers clean with ccleaner and downgraded nvidia drivers to 314.22 and its stable for now. There has been lots of talk about on forums about recent nvidia drivers, nvidia update and windows driver updates causing havoc. It could be your on board video driver causing issues. To eliminate this issue I totally disabled mine with a jumper on the motherboard. Stress testing the cards with uningine is a good idea. Stay away from fur mark though. Good luck

Wishgranter

  • Hero Member
  • *****
  • Posts: 1202
    • View Profile
    • Museum of Historic Buildings
Re: System hard freeze while GPUs processing with ATI7970 if using computer
« Reply #9 on: September 10, 2013, 08:40:31 AM »
Matt thanx for the info, im have in my system older (2-3+ months ) drivers on ATI and Nvidia. will instal the new ones and see what happen on my system..... 
Andy, PSU sems be OK with capacity......
----------------
www.mhb.sk

Matt

  • Full Member
  • ***
  • Posts: 104
    • View Profile
Re: System hard freeze while GPUs processing with ATI7970 if using computer
« Reply #10 on: September 10, 2013, 04:11:51 PM »
Hi Wishgranter,

Well I pushed my GTX card on luxmark and it wouldnt come back to idle at all and temps rose to 105 C (Glad I didnt cook it  :) ).  Now happily maxing all cards on ultra settings in pscan with no worries back on Nvidia driver 311.06 and catalyst 13.6.  They seem to be the best for my high performance mixed dual GPU system at present.

Heres the results of your statue benchmark on ultra on the new build with the above drivers.

Device 1 performance: 119.358 million samples/sec (CPU)
Device 2 performance: 591.419 million samples/sec (GeForce GTX 590)
Device 3 performance: 634.232 million samples/sec (GeForce GTX 590)
Device 4 performance: 1035.01 million samples/sec (Tahiti)
Total performance: 2380.02 million samples/sec

Thanks for the Linkedin endorsements. Chat soon

Matt
« Last Edit: September 10, 2013, 05:19:07 PM by Matt »

andyroo

  • Sr. Member
  • ****
  • Posts: 438
    • View Profile
Re: System hard freeze while GPUs processing with ATI7970 if using computer
« Reply #11 on: September 10, 2013, 08:50:15 PM »
OK I just had the same freeze while not doing anything (PS was open, but not active, not processing).

I was actually typing a forum msg and only had chrome, excel and PS open, so this tells me something h/w related is bad...

I think I am looking at a defective card or defective system memory. Will test more aggressively.

Andy

Wishgranter

  • Hero Member
  • *****
  • Posts: 1202
    • View Profile
    • Museum of Historic Buildings
Re: System hard freeze while GPUs processing with ATI7970 if using computer
« Reply #12 on: September 10, 2013, 09:23:47 PM »
Andy sometimes the problem can be very tricky to pinpoint. for now we get problem with one GTX 680 that its not working, just error 43 - and it seems that is hw realted to bad IRQ assignment :-) Sometimes it work, sometimes not.... its try and error

 try download older drivers, say 2-3 months back and install and see whats changed. My workflow is so thatif my system is running, never update Win and GPU drivers just occasionaly, every few months.... 
----------------
www.mhb.sk

andyroo

  • Sr. Member
  • ****
  • Posts: 438
    • View Profile
Re: System hard freeze while GPUs processing with ATI7970 if using computer
« Reply #13 on: September 11, 2013, 01:54:40 AM »
Talked with GIGABYTE tech support, they recommended 3DMark Vantage to test GPU. I bought it, ran it, got partway through the Jane Nash demo and got the hard freeze. That was only using one GPU, so I guess that's the bad one... Going to verify by disconnecting and running test on the other one, but thanks for all your help everyone.

Wishgranter

  • Hero Member
  • *****
  • Posts: 1202
    • View Profile
    • Museum of Historic Buildings
Re: System hard freeze while GPUs processing with ATI7970 if using computer
« Reply #14 on: September 11, 2013, 10:37:06 AM »
Matt and other, for BEST performance with multiple GPUs try DISABLE all CPU cores for processing ( observe CPU usage then, for understanding why :-) 

 Matt, if can post then again results :-D

For the AMD  7xxx ( and probably even 6xxx and 5xxx ) series see under Task Manger-PROCESS and search for this one https://www.dropbox.com/s/s9bc9s0lz96fkgm/catalyst.jpg - over time it will consume full one core and up to few (10+)  GB of ram - there is some memory overflow. not repaired a long time already. can kill it without any problems. it will rerun again but with normal memory footprint....
----------------
www.mhb.sk