Forum

Author Topic: launch error at line 67  (Read 19898 times)

FlyRight-3D

  • Newbie
  • *
  • Posts: 16
    • View Profile
launch error at line 67
« on: February 18, 2017, 07:34:48 PM »
I have been getting the failure "launch error at line 67" when performing a dense cloud generation.  it always happens during the restructuring depth phase.   I fiiiiiiiiinally got it to finish processing by selecting the lowest setting possible.  I have the necessary computing power to do 15 of these at one time, so I don't understand why I am having this trouble.   I am running dual xeon e5 2670s, 128GB RAM, and a nvidia geforce titan black.   any ideas?

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14813
    • View Profile
Re: launch error at line 67
« Reply #1 on: February 18, 2017, 09:09:18 PM »
Hello FlyRight-3D,

Are you using latest GPU drivers, also is there only NVIDIA card checked on in the GPU tab of the Preferenes window?
Best regards,
Alexey Pasumansky,
Agisoft LLC

muzykanci

  • Newbie
  • *
  • Posts: 1
    • View Profile
Re: launch error at line 67
« Reply #2 on: February 18, 2017, 10:42:50 PM »
Hi,
I had similar problem when performing a dense cloud generation:

"2017-02-16 19:04:12 CPU fallbacks: 1
2017-02-16 19:04:12 Warning: unspecified launch failure
2017-02-16 19:04:12 Warning: invalid resource handle
2017-02-16 19:04:12 Warning: unspecified launch failure
2017-02-16 19:04:12 Warning: invalid resource handle
2017-02-16 19:04:12 Warning: unspecified launch failure
2017-02-16 19:04:12 Warning: invalid resource handle
2017-02-16 19:04:12 Warning: unspecified launch failure
2017-02-16 19:04:12 Saving project...
2017-02-16 19:04:12 Warning: invalid resource handle
2017-02-16 19:04:12 Error: unspecified launch failure at line 67"

When I turned the NVIDIA card off in the GPU tab it worked fine using CPU only (much slower though)

FlyRight-3D

  • Newbie
  • *
  • Posts: 16
    • View Profile
Re: launch error at line 67
« Reply #3 on: February 19, 2017, 10:34:59 AM »
I had the nvidia card selected in the preferences; when I deselected it, I am able to do the processing.  All drivers are up to date.  There were a couple of times I had this same error during photo alignment when I was using GPU acceleration but other times I didn't.  Same with the point cloud, sometimes it would go ahead and process on a lower quality setting but it works on all settings with GPU acceleration off.  I assume it's the size of the dataset that is messing with me.  It's not particularly big and would be great if someone from agisoft could find a solution without me purchasing yet another graphics card.

Poutinemaster

  • Newbie
  • *
  • Posts: 3
    • View Profile
Re: launch error at line 67
« Reply #4 on: February 21, 2017, 07:01:19 PM »
Hello Agisoft and the community,

We are experiencing the same issue on our side.

We have 6 Gb nVidia TitanX cards and 128 GB SDRAM in each of our PCs we use for computing.  Both the Windows Task manager' memory monitoring section and GPUZ tool are reporting to us that this is not a (lack) of memory issue.

Closing Agisoft and reopening seems to be a workaround which works for us most of the time.

So far we used the driver nVidia, which is 358.50.  We tried on one PC to upgrade to 378.66, we have the exact same issue.

I added below some logs, in case it will help.

In advance, thank you!

2017-02-21 09:40:27 BuildDenseCloud: quality = High, depth filtering = Moderate
2017-02-21 09:40:27 Initializing...
2017-02-21 09:40:27 Using device: GeForce GTX TITAN Black, 15 compute units, 6144 MB global memory, CUDA 3.5
2017-02-21 09:40:27   max work group size 1024
2017-02-21 09:40:27   max work item sizes [1024, 1024, 64]
2017-02-21 09:40:27 Using CUDA device 'GeForce GTX TITAN Black' in concurrent. (2 times)
2017-02-21 09:40:27 sorting point cloud... done in 0 sec
2017-02-21 09:40:27 processing matches... done in 0.016 sec
2017-02-21 09:40:27 initializing...
2017-02-21 09:40:27 selected 34 cameras from 34 in 0.234 sec
2017-02-21 09:40:27 Loading photos...
2017-02-21 09:40:34 loaded photos in 6.625 seconds
2017-02-21 09:40:34 Reconstructing depth...
2017-02-21 09:40:36 Hierarchical SGM failed: 2017-02-21 09:40:36 GPU processing failed, switching to CPU mode
unspecified launch failure at line 67
2017-02-21 09:40:36 Hierarchical SGM failed: 2017-02-21 09:40:36 GPU processing failed, switching to CPU mode
unspecified launch failure at line 67
2017-02-21 09:40:37 [CPU] estimating 4438x3007x736 disparity using 1110x1003x16u tiles
2017-02-21 09:40:39 [CPU] estimating 4553x4737x320 disparity using 1139x948x16u tiles
2017-02-21 09:40:39 [CPU] estimating 4553x4737x320 disparity using 1139x948x16u tiles
2017-02-21 09:40:56 timings: rectify: 2.032 disparity: 19.328 borders: 0.265 filter: 0.423 fill: 0
2017-02-21 09:40:58 [CPU] estimating 4365x2929x512 disparity using 1092x977x16u tiles
2017-02-21 09:41:16 timings: rectify: 0.562 disparity: 17.782 borders: 0.359 filter: 0.469 fill: 0
2017-02-21 09:41:18 [CPU] estimating 4298x2969x544 disparity using 1075x990x16u tiles
2017-02-21 09:41:35 timings: rectify: 0.578 disparity: 17.235 borders: 0.374 filter: 0.454 fill: 0
2017-02-21 09:41:38 [CPU] estimating 4720x3027x1120 disparity using 944x1009x16u tiles
2017-02-21 09:41:57 timings: rectify: 0.64 disparity: 20.157 borders: 0.25 filter: 0.407 fill: 0
2017-02-21 09:42:00 [CPU] estimating 2869x4988x384 disparity using 957x998x16u tiles
2017-02-21 09:42:20 timings: rectify: 0.688 disparity: 21.016 borders: 0.391 filter: 0.531 fill: 0
2017-02-21 09:42:25 finished depth reconstruction in 110.957 seconds
2017-02-21 09:42:25 Device 1: 100% work done with performance: 37.7101 million samples/sec (CPU), device used for 95.518 seconds
2017-02-21 09:42:25 Device 2: 0% work done with performance: 0 million samples/sec (GeForce GTX TITAN Black), device used for 0 seconds
2017-02-21 09:42:25 Total performance: 37.7101 million samples/sec
2017-02-21 09:42:25 Warning: invalid resource handle
2017-02-21 09:42:25 Warning: invalid resource handle
2017-02-21 09:42:25 Warning: invalid resource handle
2017-02-21 09:42:25 Warning: invalid resource handle
2017-02-21 09:42:25 Finished processing in 118.082 sec (exit code 0)
2017-02-21 09:42:25 Error: unspecified launch failure at line 67

-----------------

[‎2017-‎02-‎21 09:44] Richard Côté:
No Title
2017-02-21 09:40:27 BuildDenseCloud: quality = High, depth filtering = Moderate
2017-02-21 09:40:27 Initializing...
2017-02-21 09:40:27 Using device: GeForce GTX TITAN Black, 15 compute units, 6144 MB global memory, CUDA 3.5
2017-02-21 09:40:27   max work group size 1024
2017-02-21 09:40:27   max work item sizes [1024, 1024, 64]
2017-02-21 09:40:27 Using CUDA device 'GeForce GTX TITAN Black' in concurrent. (2 times)
2017-02-21 09:40:27 sorting point cloud... done in 0 sec
2017-02-21 09:40:27 processing matches... done in 0.016 sec
2017-02-21 09:40:27 initializing...
2017-02-21 09:40:27 selected 34 cameras from 34 in 0.234 sec
2017-02-21 09:40:27 Loading photos...
2017-02-21 09:40:34 loaded photos in 6.625 seconds
2017-02-21 09:40:34 Reconstructing depth...
2017-02-21 09:40:36 Hierarchical SGM failed: 2017-02-21 09:40:36 GPU processing failed, switching to CPU mode
unspecified launch failure at line 67
2017-02-21 09:40:36 Hierarchical SGM failed: 2017-02-21 09:40:36 GPU processing failed, switching to CPU mode
unspecified launch failure at line 67
2017-02-21 09:40:37 [CPU] estimating 4438x3007x736 disparity using 1110x1003x16u tiles
2017-02-21 09:40:39 [CPU] estimating 4553x4737x320 disparity using 1139x948x16u tiles
2017-02-21 09:40:39 [CPU] estimating 4553x4737x320 disparity using 1139x948x16u tiles
2017-02-21 09:40:56 timings: rectify: 2.032 disparity: 19.328 borders: 0.265 filter: 0.423 fill: 0
2017-02-21 09:40:58 [CPU] estimating 4365x2929x512 disparity using 1092x977x16u tiles
2017-02-21 09:41:16 timings: rectify: 0.562 disparity: 17.782 borders: 0.359 filter: 0.469 fill: 0
2017-02-21 09:41:18 [CPU] estimating 4298x2969x544 disparity using 1075x990x16u tiles
2017-02-21 09:41:35 timings: rectify: 0.578 disparity: 17.235 borders: 0.374 filter: 0.454 fill: 0
2017-02-21 09:41:38 [CPU] estimating 4720x3027x1120 disparity using 944x1009x16u tiles
2017-02-21 09:41:57 timings: rectify: 0.64 disparity: 20.157 borders: 0.25 filter: 0.407 fill: 0
2017-02-21 09:42:00 [CPU] estimating 2869x4988x384 disparity using 957x998x16u tiles
2017-02-21 09:42:20 timings: rectify: 0.688 disparity: 21.016 borders: 0.391 filter: 0.531 fill: 0
2017-02-21 09:42:25 finished depth reconstruction in 110.957 seconds
2017-02-21 09:42:25 Device 1: 100% work done with performance: 37.7101 million samples/sec (CPU), device used for 95.518 seconds
2017-02-21 09:42:25 Device 2: 0% work done with performance: 0 million samples/sec (GeForce GTX TITAN Black), device used for 0 seconds
2017-02-21 09:42:25 Total performance: 37.7101 million samples/sec
2017-02-21 09:42:25 Warning: invalid resource handle
2017-02-21 09:42:25 Warning: invalid resource handle
2017-02-21 09:42:25 Warning: invalid resource handle
2017-02-21 09:42:25 Warning: invalid resource handle
2017-02-21 09:42:25 Finished processing in 118.082 sec (exit code 0)
2017-02-21 09:42:25 Error: unspecified launch failure at line 67

------------------

2017-02-21 09:45:00 Hierarchical SGM failed: 2017-02-21 09:45:00 GPU processing failed, switching to CPU mode
unspecified launch failure at line 67
2017-02-21 09:45:00 Hierarchical SGM failed: 2017-02-21 09:45:00 GPU processing failed, switching to CPU mode
unspecified launch failure at line 67

FlyRight-3D

  • Newbie
  • *
  • Posts: 16
    • View Profile
Re: launch error at line 67
« Reply #5 on: February 24, 2017, 09:54:01 AM »
ah, thanks for the workaround.  I suppose that's why it intermittently worked for me.  I rolled back the drivers a dozen versions or so and it didn't seem to help.

If need be, I will also supply log information to help with this issue.

bortoman

  • Newbie
  • *
  • Posts: 18
    • View Profile
Re: launch error at line 67
« Reply #6 on: February 28, 2017, 04:14:45 PM »
Hi, i fear this issue is not taken serious enough!

By doing some tests on 5 different computers (all with different Nvidia cards) i can reproduce the issue every time i want...  :-\

It seems to me, that you just have to go high enough with the settings, or increase the dataset for the problem to occur!
(or low enough for it not to happen - depends on your point of view)

I just did some tests with this dataset: http://www.agisoft.com/datasets/building.zip

When generating the dense cloud with ultra high settings and moderate filtering it consistently fails:

Code: [Select]
2017-02-28 12:19:42 Agisoft PhotoScan Professional Version: 1.3.0 build 3772 (64 bit)
2017-02-28 12:19:42 Platform: Windows
2017-02-28 12:19:43 OpenGL Vendor: NVIDIA Corporation
2017-02-28 12:19:43 OpenGL Renderer: GeForce GTX 580/PCIe/SSE2
2017-02-28 12:19:43 OpenGL Version: 4.5.0 NVIDIA 376.53
2017-02-28 12:19:43 Maximum Texture Size: 16384
2017-02-28 12:19:43 Quad Buffered Stereo: not enabled
2017-02-28 12:19:43 ARB_vertex_buffer_object: supported
2017-02-28 12:19:43 ARB_texture_non_power_of_two: supported
2017-02-28 12:19:43 LoadProject
2017-02-28 12:19:43 Loading project...
2017-02-28 12:19:45 loaded project in 2.124 sec
2017-02-28 12:19:45 Finished processing in 2.124 sec (exit code 1)
2017-02-28 12:20:16 Checking for missing images...
2017-02-28 12:20:16 checking for missing images... done in 0.016 sec
2017-02-28 12:20:16 Finished processing in 0.016 sec (exit code 1)
2017-02-28 12:20:16 BuildDenseCloud: quality = Ultra high, depth filtering = Moderate
2017-02-28 12:20:16 Initializing...
2017-02-28 12:20:16 Using device: GeForce GTX 580, 16 compute units, 1536 MB global memory, CUDA 2.0
2017-02-28 12:20:16   max work group size 1024
2017-02-28 12:20:16   max work item sizes [1024, 1024, 64]
2017-02-28 12:20:16 sorting point cloud... done in 0 sec
2017-02-28 12:20:16 processing matches... done in 0 sec
2017-02-28 12:20:16 initializing...
2017-02-28 12:20:16 selected 50 cameras from 50 in 0.125 sec
2017-02-28 12:20:16 Loading photos...
2017-02-28 12:20:20 loaded photos in 4.312 seconds
2017-02-28 12:20:20 Reconstructing depth...
2017-02-28 12:20:22 [GPU] estimating 3498x2857x1728 disparity using 1166x1429x8u tiles
2017-02-28 12:20:25 [CPU] estimating 4002x2665x1120 disparity using 1001x889x8u tiles
2017-02-28 12:20:32 timings: rectify: 1.891 disparity: 8.733 borders: 0.579 filter: 0.655 fill: 0
2017-02-28 12:20:35 [GPU] estimating 2795x2800x224 disparity using 466x400x8u tiles
2017-02-28 12:20:43 timings: rectify: 1.157 disparity: 6.5 borders: 0.546 filter: 0.548 fill: 0
2017-02-28 12:20:45 [GPU] estimating 3561x2717x1024 disparity using 1187x906x8u tiles
2017-02-28 12:20:55 timings: rectify: 1.234 disparity: 8.126 borders: 0.562 filter: 0.672 fill: 0
2017-02-28 12:21:01 timings: rectify: 1.953 disparity: 36.733 borders: 0.702 filter: 0.766 fill: 0
2017-02-28 12:21:01 [GPU] estimating 3961x2493x416 disparity using 793x831x8u tiles
2017-02-28 12:21:05 [CPU] estimating 3498x2857x1728 disparity using 1166x953x8u tiles
2017-02-28 12:21:10 timings: rectify: 1.437 disparity: 8.006 borders: 0.781 filter: 0.812 fill: 0
2017-02-28 12:21:14 [GPU] estimating 4002x2665x1120 disparity using 1001x889x8u tiles
2017-02-28 12:21:24 timings: rectify: 1.797 disparity: 9.003 borders: 0.515 filter: 0.579 fill: 0
2017-02-28 12:21:32 [GPU] estimating 2412x12895x512 disparity using 804x1173x8u tiles
2017-02-28 12:21:41 timings: rectify: 0.969 disparity: 38.007 borders: 0.344 filter: 0.447 fill: 0
2017-02-28 12:21:47 [CPU] estimating 4207x1867x512 disparity using 1052x934x8u tiles
2017-02-28 12:21:58 timings: rectify: 5.922 disparity: 20.824 borders: 2.357 filter: 3.076 fill: 0
2017-02-28 12:22:04 [GPU] estimating 3980x1867x448 disparity using 796x934x8u tiles
2017-02-28 12:22:12 timings: rectify: 1.193 disparity: 6.464 borders: 0.743 filter: 0.767 fill: 0
2017-02-28 12:22:15 [GPU] estimating 4116x1809x800 disparity using 824x905x8u tiles
2017-02-28 12:22:16 timings: rectify: 0.881 disparity: 29.64 borders: 0.489 filter: 0.58 fill: 0
2017-02-28 12:22:20 [CPU] estimating 4143x2414x640 disparity using 1036x1207x8u tiles
2017-02-28 12:22:23 timings: rectify: 1.038 disparity: 6.278 borders: 0.709 filter: 0.667 fill: 0
2017-02-28 12:22:25 [GPU] estimating 4040x1907x832 disparity using 808x954x8u tiles
2017-02-28 12:22:34 timings: rectify: 1.186 disparity: 7.203 borders: 0.643 filter: 0.533 fill: 0
2017-02-28 12:22:37 [GPU] estimating 4091x2108x960 disparity using 1023x703x8u tiles
2017-02-28 12:22:45 timings: rectify: 1.037 disparity: 7.22 borders: 0.624 filter: 0.765 fill: 0
2017-02-28 12:22:48 [GPU] estimating 4028x1981x736 disparity using 806x991x8u tiles
2017-02-28 12:22:55 timings: rectify: 1.049 disparity: 35.688 borders: 0.664 filter: 0.706 fill: 0
2017-02-28 12:22:56 timings: rectify: 1.219 disparity: 6.545 borders: 0.825 filter: 0.873 fill: 0
2017-02-28 12:22:59 [GPU] estimating 3948x1898x288 disparity using 790x949x8u tiles
2017-02-28 12:23:01 [CPU] estimating 3962x2493x416 disparity using 991x1247x8u tiles
2017-02-28 12:23:06 timings: rectify: 0.921 disparity: 5.268 borders: 0.828 filter: 0.754 fill: 0
2017-02-28 12:23:15 [GPU] estimating 4116x1809x800 disparity using 824x905x8u tiles
2017-02-28 12:23:23 timings: rectify: 1.707 disparity: 6.782 borders: 0.671 filter: 0.678 fill: 0
2017-02-28 12:23:26 [GPU] estimating 4090x1982x480 disparity using 818x991x8u tiles
2017-02-28 12:23:34 timings: rectify: 1.409 disparity: 6.47 borders: 0.81 filter: 0.794 fill: 0
2017-02-28 12:23:37 timings: rectify: 2.085 disparity: 36.377 borders: 0.946 filter: 0.942 fill: 0
2017-02-28 12:23:37 [GPU] estimating 4143x2414x640 disparity using 1036x805x8u tiles
2017-02-28 12:23:40 [CPU] estimating 3502x2089x192 disparity using 1168x1045x8u tiles
2017-02-28 12:23:49 timings: rectify: 1.538 disparity: 10.187 borders: 0.973 filter: 1.083 fill: 0
2017-02-28 12:23:52 [GPU] estimating 4201x1868x864 disparity using 841x934x8u tiles
2017-02-28 12:24:01 timings: rectify: 1.237 disparity: 7.529 borders: 0.5 filter: 0.596 fill: 0
2017-02-28 12:24:04 [GPU] estimating 4124x1900x800 disparity using 825x950x8u tiles
2017-02-28 12:24:08 timings: rectify: 0.842 disparity: 28.139 borders: 0.53 filter: 0.564 fill: 0
2017-02-28 12:24:13 timings: rectify: 1.431 disparity: 7.653 borders: 0.696 filter: 0.611 fill: 0
2017-02-28 12:24:15 [GPU] estimating 3581x1597x384 disparity using 1194x799x8u tiles
2017-02-28 12:24:16 [CPU] estimating 2795x2800x224 disparity using 932x934x8u tiles
2017-02-28 12:24:22 timings: rectify: 0.728 disparity: 5.365 borders: 0.58 filter: 0.569 fill: 0
2017-02-28 12:24:30 [GPU] estimating 3730x2433x1216 disparity using 933x811x8u tiles
2017-02-28 12:24:40 timings: rectify: 1.169 disparity: 8.583 borders: 0.5 filter: 0.593 fill: 0
2017-02-28 12:24:43 [GPU] estimating 4028x2848x800 disparity using 806x950x8u tiles
2017-02-28 12:24:46 timings: rectify: 1.415 disparity: 30.671 borders: 0.498 filter: 0.649 fill: 0
2017-02-28 12:24:51 [CPU] estimating 3816x3128x576 disparity using 954x1043x8u tiles
2017-02-28 12:24:57 timings: rectify: 1.963 disparity: 11.251 borders: 0.792 filter: 1.165 fill: 0
2017-02-28 12:25:00 [GPU] estimating 4038x2330x640 disparity using 1010x777x8u tiles
2017-02-28 12:25:11 timings: rectify: 1.58 disparity: 8.472 borders: 0.699 filter: 0.992 fill: 0
2017-02-28 12:25:14 [GPU] estimating 3883x2433x768 disparity using 971x811x8u tiles
2017-02-28 12:25:24 timings: rectify: 1.379 disparity: 8.451 borders: 0.679 filter: 0.779 fill: 0
2017-02-28 12:25:33 [GPU] estimating 3561x2717x1024 disparity using 891x906x8u tiles
2017-02-28 12:25:38 timings: rectify: 1.52 disparity: 48.304 borders: 0.687 filter: 0.833 fill: 0
2017-02-28 12:25:43 [CPU] estimating 3730x2433x1216 disparity using 933x1217x8u tiles
2017-02-28 12:25:43 timings: rectify: 1.659 disparity: 9.446 borders: 0.548 filter: 0.612 fill: 0
2017-02-28 12:25:47 [GPU] estimating 3816x3128x576 disparity using 954x782x8u tiles
2017-02-28 12:25:59 timings: rectify: 1.868 disparity: 10.832 borders: 0.711 filter: 0.834 fill: 0
2017-02-28 12:26:02 [GPU] estimating 4028x2848x800 disparity using 1007x950x8u tiles
2017-02-28 12:26:14 timings: rectify: 1.383 disparity: 10.184 borders: 0.832 filter: 1.024 fill: 0
2017-02-28 12:26:18 [GPU] estimating 4084x2745x1088 disparity using 817x687x8u tiles
2017-02-28 12:26:18 timings: rectify: 1.129 disparity: 36.451 borders: 0.469 filter: 0.632 fill: 0
2017-02-28 12:26:26 [CPU] estimating 4083x2745x1088 disparity using 1021x915x8u tiles
2017-02-28 12:26:32 timings: rectify: 1.767 disparity: 12.628 borders: 0.494 filter: 0.689 fill: 0
2017-02-28 12:26:40 [GPU] estimating 3884x2433x768 disparity using 971x811x8u tiles
2017-02-28 12:26:51 timings: rectify: 1.733 disparity: 8.598 borders: 0.602 filter: 0.874 fill: 0
2017-02-28 12:26:54 [GPU] estimating 3885x2857x448 disparity using 972x953x8u tiles
2017-02-28 12:27:05 timings: rectify: 1.681 disparity: 9.282 borders: 0.97 filter: 1.075 fill: 0
2017-02-28 12:27:08 timings: rectify: 1.608 disparity: 42.767 borders: 0.494 filter: 1.025 fill: 0
2017-02-28 12:27:09 [GPU] estimating 4041x2742x1184 disparity using 809x914x8u tiles
2017-02-28 12:27:12 [CPU] estimating 4038x2330x640 disparity using 1010x1165x8u tiles
2017-02-28 12:27:21 timings: rectify: 2.009 disparity: 10.629 borders: 0.734 filter: 0.863 fill: 0
2017-02-28 12:27:24 [GPU] estimating 3938x2525x1984 disparity using 985x842x8u tiles
2017-02-28 12:27:35 timings: rectify: 1.736 disparity: 9.481 borders: 0.454 filter: 0.596 fill: 0
2017-02-28 12:27:42 [GPU] estimating 3980x1867x448 disparity using 796x934x8u tiles
2017-02-28 12:27:45 timings: rectify: 1.03 disparity: 34.155 borders: 0.55 filter: 0.595 fill: 0
2017-02-28 12:27:49 timings: rectify: 1.151 disparity: 5.721 borders: 0.582 filter: 0.647 fill: 0
2017-02-28 12:27:51 [GPU] estimating 4090x1982x480 disparity using 818x991x8u tiles
2017-02-28 12:27:52 [CPU] estimating 3884x2857x448 disparity using 971x953x8u tiles
2017-02-28 12:28:01 timings: rectify: 0.883 disparity: 8.134 borders: 0.807 filter: 0.799 fill: 0
2017-02-28 12:28:04 [GPU] estimating 4207x1867x512 disparity using 842x623x8u tiles
2017-02-28 12:28:09 Hierarchical SGM failed: 2017-02-28 12:28:09 GPU processing failed, switching to CPU mode
unspecified launch failure at line 208
2017-02-28 12:28:10 [CPU] estimating 4207x1867x512 disparity using 1052x934x8u tiles
2017-02-28 12:28:28 timings: rectify: 1.242 disparity: 38.327 borders: 0.637 filter: 0.788 fill: 0
2017-02-28 12:28:31 [CPU] estimating 3461x2536x768 disparity using 1154x1268x8u tiles
2017-02-28 12:28:58 timings: rectify: 0.899 disparity: 27.291 borders: 0.48 filter: 0.545 fill: 0
2017-02-28 12:29:03 finished depth reconstruction in 522.192 seconds
2017-02-28 12:29:03 Device 1: 19% work done with performance: 16.3823 million samples/sec (CPU), device used for 462.55 seconds
2017-02-28 12:29:03 Device 2: 80% work done with performance: 68.4154 million samples/sec (GeForce GTX 580), device used for 275.848 seconds
2017-02-28 12:29:03 Total performance: 84.7977 million samples/sec
2017-02-28 12:29:03 Warning: unspecified launch failure
2017-02-28 12:29:03 Warning: invalid resource handle
2017-02-28 12:29:03 Finished processing in 527.177 sec (exit code 0)
2017-02-28 12:29:03 Error: unspecified launch failure at line 67

System specs for this test:
Intel I7 920
GTX 580
12GB RAM

I know this is a low-spec-pc, but i'm out of time testing on our production machines...
I assure you even our dual-xeon, nvidia quadro workhorse has the same issue!

Needless to say it works fine once i reinstalled the previous version 1.2.6

I strongly suggest a rollback or at least a switch in the settings to use OpenCL again instead of CUDA!

Chernabog

  • Newbie
  • *
  • Posts: 4
    • View Profile
Re: launch error at line 67
« Reply #7 on: March 01, 2017, 02:12:51 PM »
Hi All,

Just to say that I too am experiencing this same issue trying to stich about 200 36 MP images using the Photoscan version 1.3.0.

Our system spec is as below:
18 virtual processor Xeon E5-2660 v3 (2.6 Ghz)
112 GB RAM
Nvidia GRID K2 (8 cores @ 745 Mhz, 4096 MB vRAM)

Works fine using CPU only, just takes an age to complete.

Regards,

S.

bortoman

  • Newbie
  • *
  • Posts: 18
    • View Profile
Re: launch error at line 67
« Reply #8 on: March 01, 2017, 08:40:09 PM »
Hi,
just to let you know about the progress:

Agisoft is apparently addressing the issue...
Alexey has sent me (and probably some other users) a temporary build 3884 which seems to have multiple different issues...
(I will not discuss them here, as a temporary build is not an official release, i discuss this via private message with Alexey)

All i can say, is that now i don't get an error message, but instead my computer crashes and reboots...  :o

Can someone please share his experience?

Hans Keesom

  • Newbie
  • *
  • Posts: 15
    • View Profile
Re: launch error at line 67
« Reply #9 on: March 04, 2017, 12:12:34 PM »
Isn't this the same as http://www.agisoft.com/forum/index.php?topic=6565.0

I had to go back to previous version because of this. It's a pity as 1.3.0. seems to use the GPU much more and 1080 cards are dropping in price the last few days.

Hans Keesom

  • Newbie
  • *
  • Posts: 15
    • View Profile
Re: launch error at line 67
« Reply #10 on: March 06, 2017, 12:45:51 AM »
I think I have it solved by doing a rollback of the GPU driver as suggested by someone here. See http://www.nvidia.com/object/driver_rollback.html for how to

Now running with GPU average of 58% version 1.3.0 instead of 41% with version 1.2.6. !

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14813
    • View Profile
Re: launch error at line 67
« Reply #11 on: March 06, 2017, 01:11:55 AM »
Hello Hans,

Which driver version you have rolled back to?
Best regards,
Alexey Pasumansky,
Agisoft LLC

FlyRight-3D

  • Newbie
  • *
  • Posts: 16
    • View Profile
Re: launch error at line 67
« Reply #12 on: March 06, 2017, 07:05:51 PM »
I didn't have luck rolling back the drivers but I only tried a few. 

So here is where I am now with these large datasets 200-500 photographs at 12-20MP; I have had luck throttling back the clock speed of my GPU.  I was getting performance caps based on available power and voltage reliability on my Titan black.  After clocked it back a bit, I didn't have nearly as much trouble as before but the problem still perisisted.  I added a second, very small card out of another computer (gtx 745) to run my displays.  This also worked with returning the clock speed on my Titan black to stock settings. I am currently having success with running both GPUs during photo alignment and dense cloud generation.

From these experiences it seems that if your GPU throttles back during a heavy workload, these power fluctuations are causing some issue at "line 67".  I suggest slowing your GPU clock speed and memory clock a bit and see if this helps you.  Also try adding a monitor display card to use your larger GPU (array) to lower the demand on a single graphics card.

I hope this helps and I'll continue to update you all as I find "fixes"

FlyRight-3D

  • Newbie
  • *
  • Posts: 16
    • View Profile
Re: launch error at line 67
« Reply #13 on: March 06, 2017, 07:46:47 PM »
Also, I liquid cooled everything and dropped the temps from about 80C to 50C.  I'm not sure if heat throttling had anything to do with this or not but I figured it's worth mentioning.
« Last Edit: March 06, 2017, 10:21:44 PM by FlyRight-3D »

Hans Keesom

  • Newbie
  • *
  • Posts: 15
    • View Profile
Re: launch error at line 67
« Reply #14 on: March 07, 2017, 01:15:31 AM »
Hello Hans,

Which driver version you have rolled back to?

Sorry, the error did show up again in align photos. I was too happy too soon :-(
GPU crashes on Allign photos still

I now run/test allign photos with GPU processing off and the rest with it on, see how that goes
« Last Edit: March 07, 2017, 01:43:56 AM by Hans Keesom »