UI not responding on dense cloud/align photos

[PaleoPixels]

Hi all,

Sometimes, especially on larger scenes with 2000+ photos, the UI stops responding. This is on all 3 Photoscan (not pro) machines, (all 2x5675 Xeon, each has dual GPU). When the UI stops giving feedback, taskmanager says the CPU's are still being used, memory usage changes too over time.
After around 10 hours I manually stopped Photoscan and tried again on all machines. Same behavior , 1 on building dense cloud, other on estimating camera positions.
Question : is this normal behavior and should I just let Photoscan continue or has the programm crashed (though still using cpu cycles) ?

Thank you,
Bjorn

[Alexey Pasumansky]

Hello Bjorn,

Which version of PhotoScan you are using and can you also provide full processing log related to such operation?

[PaleoPixels]

Hi Alexey,

I'll send you the complete logs in a pm.
All 3 computers crashed after running them from yesterday afternoon to this morning.
The last pieces of each log:

pc 2
2017-04-20 14:25:08 points detected in 317.411 sec
2017-04-20 14:25:08 Selecting pairs...
2017-04-20 14:25:08 Using device: GeForce GTX 1050 Ti, 6 compute units, 4096 MB global memory, Device Compute Capability 6.1
2017-04-20 14:25:08   max work group size 1024
2017-04-20 14:25:08   max work item sizes [1024, 1024, 64]
2017-04-20 14:25:08 Using device: GeForce GTX 1060 6GB, 10 compute units, 6144 MB global memory, Device Compute Capability 6.1
2017-04-20 14:25:08   max work group size 1024
2017-04-20 14:25:08   max work item sizes [1024, 1024, 64]

pc 3
2017-04-20 13:50:41 adjusting: xxxxxxxxxxxxxxxxxxxx 0.513683 -> 0.472332
2017-04-20 13:57:26 adding 16057 points, 1165 far (4.375 threshold), 9725 inaccurate, 5946 invisible, 465 weak
2017-04-20 13:57:26 optimized in 404.731 seconds
2017-04-20 13:57:26 f 1089.76, cx -5.11358, cy 4.86137, k1 -0.0230265, k2 0, k3 0
2017-04-20 13:57:26 f 1081.02, cx 0, cy 0, k1 -0.023253, k2 0, k3 0
2017-04-20 13:57:26 adjusting: xxxxxxxxxxxxxxxxxxxx 0.629632 -> 0.467149
2017-04-20 14:04:08 loaded projections in 0.052 sec
2017-04-20 14:04:08 tracks initialized in 0.372 sec
2017-04-20 14:04:08 adding 146857 points, 1178 far (4.375 threshold), 8190 inaccurate, 6006 invisible, 473 weak
2017-04-20 14:04:08 adding camera 0 (890 of 2432), 211 of 268 used
2017-04-20 14:04:08 adding camera 0 (891 of 2432), 154 of 208 used
2017-04-20 14:04:08 adding camera 0 (892 of 2432), 148 of 163 used
2017-04-20 14:04:08 adding camera 0 (893 of 2432), 137 of 196 used
2017-04-20 14:04:08 adding 16439 points, 1238 far (4.375 threshold), 8212 inaccurate, 6047 invisible, 472 weak
2017-04-20 14:04:09 block: 1 sensors, 1071 cameras, 183975 points, 657336 projections
2017-04-20 14:04:09 block_sensors: 0.000427246 MB (0.000427246 MB allocated)
2017-04-20 14:04:09 block_cameras: 0.335014 MB (0.335014 MB allocated)
2017-04-20 14:04:09 block_points: 8.42171 MB (9.49319 MB allocated)
2017-04-20 14:04:09 block_tracks: 0.701809 MB (0.701809 MB allocated)
2017-04-20 14:04:09 block_obs: 12.5377 MB (12.5377 MB allocated)
2017-04-20 14:04:09 block_ofs: 1.40363 MB (1.40363 MB allocated)
2017-04-20 14:04:09 block_fre: 0.498962 MB (0.498962 MB allocated)
2017-04-20 14:04:09 adding 156087 points, 1576 far (4.375 threshold), 8170 inaccurate, 5708 invisible, 832 weak
2017-04-20 14:04:09 adjusting: xxxxxxxxxxxxxxxxxxxx 0.545151 -> 0.518666
2017-04-20 14:26:16 adding 16294 points, 1589 far (4.375 threshold), 8977 inaccurate, 5716 invisible, 842 weak
2017-04-20 14:26:16 optimized in 1327.87 seconds
2017-04-20 14:26:16 f 1086.18, cx -6.36199, cy 9.55988, k1 -0.0233555, k2 1.4651e-007, k3 0
2017-04-20 14:26:17 adjusting: xxxxxxxxx


pc 1
2017-04-21 07:42:47 block_fre: 0.161228 MB (0.161228 MB allocated)
2017-04-21 07:42:47 adding 480 points, 0 far (4.375 threshold), 0 inaccurate, 0 invisible, 0 weak
2017-04-21 07:42:47 adjusting: xxxxxx 0.65155 -> 0.644389
2017-04-21 07:42:47 adding 0 points, 0 far (4.375 threshold), 0 inaccurate, 0 invisible, 0 weak
2017-04-21 07:42:47 optimized in 0.016 seconds
2017-04-21 07:42:47 f 1083.4, cx 0, cy 0, k1 0, k2 0, k3 0
2017-04-21 07:42:47 adjusting: xx 0.644389 -> 0.644389
2017-04-21 07:42:47 loaded projections in 0.024 sec
2017-04-21 07:42:47 tracks initialized in 0.171 sec
2017-04-21 07:42:47 adding 480 points, 0 far (4.375 threshold), 0 inaccurate, 0 invisible, 0 weak
2017-04-21 07:42:47 2 blocks: 2894 27
2017-04-21 07:42:47 calculating match counts... done in 0.017 sec
2017-04-21 07:42:47 Not enough points to align
2017-04-21 07:42:47 groups 0 and 1: 0 robust from 1
2017-04-21 07:42:47 groups 1 and 0: 0 robust from 1
2017-04-21 07:42:47 overlapping groups selected in 0.025 sec
2017-04-21 07:42:47 2 blocks: 2894 27
2017-04-21 07:42:48 final block size: 2894
2017-04-21 07:42:49 adding 1539019 points, 7508 far (4.375 threshold), 79659 inaccurate, 19264 invisible, 2353 weak
2017-04-21 07:42:49 3 sigma filtering...
2017-04-21 07:42:50 adjusting: xxxxxxxx




Is there anything I can do? Is there a certain videocard driver recommended? Is there a version of Photoscan possible that outputs more info in the log?

Thank you

Bjorn

[PaleoPixels]

System info:

pc 1
HP Z800
2x Xeon 5675
Geforce 970 + Geforce 1050ti
Driver 376.09
96 GB ram
Windows 7 Pro with all updates installed

pc 2
HP Z800
2x Xeon 5675
Geforce 1060 + Geforce 1050ti
Driver 375.70
32 GB ram
Windows 7 Pro with all updates installed

pc 3
HP Z800
2x Xeon 5675
Geforce 1050ti + Geforce 1050ti
Driver 375.70
32 GB ram
Windows 7 Pro with all updates installed

[Alexey Pasumansky]

Hello Bjorn,

The first crash may be related to the GPU driver, as the processing stopped on GPU task. Other crashes were during CPU only tasks, so it could be a memory issue. So it may be reasonable to use Memtest86 to scan memory on all the computers. Also check that there's no overheat issue.
As for GPU driver, it's better to install the latest version from NVIDIA web-site.

Were there crash reporter tool displayed? If it is shown after such issues, please submit the crash report.

[PaleoPixels]

Hi Alexey,

I updated my Nvidia  drivers on all computers to 381.65 which is the latest on the Nvidia website. I also disabled 4 of the 24 cpu for Photoscan.
It's been running for 25 hours now (dense cloud generation , 1472 photo's,high quality) and so far it has not crashed....so that's good I have the feeling though it's (1.3.1) slower as 1.2.6 .

though GPU acceleration is enabled, the GPU have clocked themselves down (clockspeed) and are not used, only 1 CPU thread so that may explain the long time it takes?

-Bjorn

[Alexey Pasumansky]

Hello Bjorn,

What do you mean by disabling 4 out of 24 cores? And do you have writing of the processing log to the file enabled in Preferences, so that we can also check the log?

[PaleoPixels]

Yes I send you a pm with 3 logs. With disabling CPU, I mean Windows ->task manager->set affinity ->disable 4 out of 24 cores there).

[Alexey Pasumansky]

And another  question, do you have Service Pack 1 installed on your Windows 7 systems?

[PaleoPixels]

Yes, all systems are updated.

[PaleoPixels]

After running this afternoon on 3 pc's with 3 jobs:

1 job , dense cloud generation:
 1 still going on 1 CPU thead which will be done tomorrow,gpu not used on 'generating dense point cloud'

1 job,also dense cloud:
1 using all CPU threads, will be done in 4 hours, gpu not used on 'generating dense point cloud'

1 job still going on 1 cpu thread 'estimating camera locations', done in 56 hours


So a large speed difference between individual photoscan instances, somehow photoscan decides to only use 1 CPU thread instead of all threads available?

-Bjorn

[Alexey Pasumansky]

Hello Bjorn,

Just to confirm:
- other processing stages were using all available cores?
- there's plenty of free RAM on the machines using only one CPU core and swap is not used?

Also if you can save the project in PSZ format after alignment (no need of the dense cloud and depth maps) that has been processing for a long time, and send it to us, we'll try to reproduce the problem on a similar configuration based on the matching points stored in the project.