Agisoft Metashape

Agisoft Metashape => Bug Reports => Topic started by: zmeski on June 19, 2017, 10:32:00 AM

Title: Calculating vertex colors - stuck
Post by: zmeski on June 19, 2017, 10:32:00 AM

Hi guys,
using the latest version on W10, i started having issues when calculating vertex colors
it stays at 0%
processor load is 1%
and it sticks there foreverHappens with every model i do.
Any idea when this will be sorted? It clearly is a bug.
Cheers
Martin

Title: Re: Calculating vertex colors - stuck
Post by: Alexey Pasumansky on June 19, 2017, 11:08:02 AM

Hello Martin,

Can you please specify the number of points in the dense cloud, number of the images used and image resolution and also the processing settings utilized in Build Mesh dialog?

Also please check the memory consumption at the moment and commit level (usually low CPU usage indicates the insufficient RAM, when disk swap is used).

Note that in the version 1.3.2 it is possible to disable mesh vertex colors calculation.

Title: Re: Calculating vertex colors - stuck
Post by: zmeski on June 19, 2017, 02:11:44 PM

Hello!
Its 106 photos.
Dense cloud has almost 14 million points
My computer has 256GB of RAM so thats more than enough.
It only started doing recently, in previous versions it was fine.
But i have to say i have started seeing a lot processor usage drops too along the creation.
I have 2xeons with 56 cores altogether at 3 Ghz
M

Title: Re: Calculating vertex colors - stuck
Post by: Alexey Pasumansky on June 19, 2017, 02:20:52 PM

Hello Martin,

Which mesh reconstruction option you are using (Arbitrary or Height Field) and can you provide the processing log related to the operation?

Title: Re: Calculating vertex colors - stuck
Post by: zmeski on June 19, 2017, 03:08:32 PM

i am using arbitrary.
I will provide the log info when this happens again.
I am currently having to omit the vertex color calculation as i am under deadline with the model.
will post soon hopefully,
thanks for the support!

Title: Re: Calculating vertex colors - stuck
Post by: zmeski on June 29, 2017, 02:34:43 PM

so it happened again...where do i find the processing log please?
also ..is graphics card responsible for the vertex color count or the CPUs?

cheers

Title: Re: Calculating vertex colors - stuck
Post by: Alexey Pasumansky on June 29, 2017, 02:50:11 PM

Hello Martin,

If you have redirected the output to the external file, you can find it under the path specified in Write Log to File section of the Preferences window.
Otherwise it will be only accessible via Console pane.

Can you please also check the CPU usage, memory consumption and commit level now?

Title: Re: Calculating vertex colors - stuck
Post by: zmeski on June 29, 2017, 03:23:01 PM

CPU USAGE 0,6-1,6 %
MEM USAGE 1GB roughly

console log:
2017-06-29 12:55:30 Agisoft PhotoScan Standard Version: 1.3.2 build 4205 (64 bit)
2017-06-29 12:55:30 Platform: Windows
2017-06-29 12:55:31 OpenGL Vendor: NVIDIA Corporation
2017-06-29 12:55:31 OpenGL Renderer: GeForce GTX TITAN/PCIe/SSE2
2017-06-29 12:55:31 OpenGL Version: 4.5.0 NVIDIA 382.33
2017-06-29 12:55:31 Maximum Texture Size: 16384
2017-06-29 12:55:31 Quad Buffered Stereo: not enabled
2017-06-29 12:55:31 ARB_vertex_buffer_object: supported
2017-06-29 12:55:31 ARB_texture_non_power_of_two: supported
2017-06-29 12:55:53 LoadProject
2017-06-29 12:55:53 Loading project...
2017-06-29 12:55:53 loaded project in 0.311 sec
2017-06-29 12:55:53 Finished processing in 0.313 sec (exit code 1)
2017-06-29 13:11:03 SaveProject
2017-06-29 13:11:03 Saving project...
2017-06-29 13:11:05 saved project in 2.587 sec
2017-06-29 13:11:05 Finished processing in 2.593 sec (exit code 1)
2017-06-29 13:12:10 Analyzing dense cloud...
2017-06-29 13:12:30 Finished processing in 20.053 sec (exit code 1)
2017-06-29 13:13:15 BuildModel: surface type = Arbitrary, source data = Dense cloud, face count = High, interpolation = Enabled
2017-06-29 13:13:35 Generating mesh...
2017-06-29 13:13:35 Grid size: 2877 x 2610 x 990
2017-06-29 13:13:35 Tree depth: 12
2017-06-29 13:15:17 Tree set in 101.332s (8614073 points)
2017-06-29 13:15:18 Leaves/Nodes: 50498939/57713073
2017-06-29 13:15:25 Laplacian constraints set in 6.688s
2017-06-29 13:15:25 Depth[0/12]: 1
2017-06-29 13:15:25    Evaluated / Got / Solved in: 0 / 0 / 0.000999928
2017-06-29 13:15:25 Depth[1/12]: 8
2017-06-29 13:15:25    Evaluated / Got / Solved in: 0 / 0 / 0.00300002
2017-06-29 13:15:25 Depth[2/12]: 64
2017-06-29 13:15:25    Evaluated / Got / Solved in: 0 / 0.000999928 / 0.00399995
2017-06-29 13:15:25 Depth[3/12]: 512
2017-06-29 13:15:25    Evaluated / Got / Solved in: 0 / 0.00200009 / 0.00900006
2017-06-29 13:15:25 Depth[4/12]: 4096
2017-06-29 13:15:25    Evaluated / Got / Solved in: 0 / 0.00499988 / 0.026
2017-06-29 13:15:25 Depth[5/12]: 32768
2017-06-29 13:15:25    Evaluated / Got / Solved in: 0 / 0.0209997 / 0.0490003
2017-06-29 13:15:25 Depth[6/12]: 20432
2017-06-29 13:15:25    Evaluated / Got / Solved in: 0 / 0.0180001 / 0.0809999
2017-06-29 13:15:25 Depth[7/12]: 67848
2017-06-29 13:15:26    Evaluated / Got / Solved in: 0 / 0.0530012 / 0.158999
2017-06-29 13:15:26 Depth[8/12]: 274112
2017-06-29 13:15:26    Evaluated / Got / Solved in: 0 / 0.176999 / 0.363001
2017-06-29 13:15:26 Depth[9/12]: 1228992
2017-06-29 13:15:28    Evaluated / Got / Solved in: 0 / 0.705001 / 0.942999
2017-06-29 13:15:28 Depth[10/12]: 4889792
2017-06-29 13:15:34    Evaluated / Got / Solved in: 0 / 2.934 / 2.626
2017-06-29 13:15:34 Depth[11/12]: 16139608
2017-06-29 13:15:50    Evaluated / Got / Solved in: 0 / 8.235 / 7.269
2017-06-29 13:15:50 Depth[12/12]: 35054840
2017-06-29 13:16:23    Evaluated / Got / Solved in: 0 / 16.483 / 14.469
2017-06-29 13:16:23 Linear system solved in 57.82s
2017-06-29 13:16:24 Got Iso-value in 1.608s
2017-06-29 13:16:24 Iso-Value 1.34868
2017-06-29 13:18:28 18756342 faces extracted in 123.901s
2017-06-29 13:19:08 decimating mesh (18756342 -> 1722814)
2017-06-29 13:19:08 Decimating mesh...
2017-06-29 13:20:44 Calculating vertex colors...
2017-06-29 13:20:45 processing nodes...  done in 20.352 sec
2017-06-29 13:21:05 calculating colors... Finished processing in 3554.51 sec (exit code 0)
2017-06-29 14:12:30 Error: Aborted by user

had to abort as i could not wait longer

Title: Re: Calculating vertex colors - stuck
Post by: zmeski on June 29, 2017, 03:23:29 PM

unsure what commit level means?

Title: Re: Calculating vertex colors - stuck
Post by: Alexey Pasumansky on June 29, 2017, 03:27:06 PM

Hello Martin,

If you start the same project again for the mesh generation, would it be possible to reproduce the issue again?

Title: Re: Calculating vertex colors - stuck
Post by: zmeski on June 29, 2017, 03:43:46 PM

yes definitelly but if i dotn abort it, it will compute forever :)
so is there anything special i shoudl do this time?

Title: Re: Calculating vertex colors - stuck
Post by: zmeski on June 29, 2017, 04:23:43 PM

so i have it running now
started calculating vertex colors at 15:21:06
procesing nodes.....done
and finished now....

i have to say i have changed the graphics card for a better one
i had TITAN X (9series 6gb)
and switched to 1080TI 11GB

and it finally works

coudl it be because of graphics card and /or drivers?

Title: Re: Calculating vertex colors - stuck
Post by: Alexey Pasumansky on June 30, 2017, 12:18:21 AM

Hello Martin,

Mesh calculation is performed on CPU only.

I think that slow processing time maybe somehow related to the dual-Xeon system running Windows. You may need to adjust NODE Interleaving / NUMA Group Size Optimization parameter in BIOS.

Title: Re: Calculating vertex colors - stuck
Post by: zmeski on July 03, 2017, 10:37:58 AM

Hi Alex,

yes it started doing again :-/

But it was working fine in previous versions even on this machine so its very curious why it became behaving like this???
Perhaps some stupid Windows 10 update again?

Speaking of NODE interleaving/NUMA group size.....unfortunately i have no clue how to set it up.
I am doing lots of rendering in Vray and other computations which work just fine....so i am really afraid to mess with it.

Is there a safe tutorial how to set it uP please?

Cheers
M

Title: Re: Calculating vertex colors - stuck
Post by: zmeski on July 03, 2017, 11:00:06 AM

ok so the only setting i have in the BIOS is NUMA enable/disable
since i have dual XEON system, it shoudl be enabled.
ALL my other software always identify 2 nodes with 28 threads each.

Photoscan was working great in a previous version so somethign must have happend either on Agisoft end or in windows.

Is there a safe way to test this?

Its true that in the pas time, i coul see some processor thread usage setup in the Photoscan options.

I have also tested Reality Capture and it works withtou any delay as well as all my rendergin packages.

I really love Photoscan but whats happening now is a show stopper for me.

Any clues please?

M

Title: Re: Calculating vertex colors - stuck
Post by: zmeski on July 03, 2017, 11:17:50 AM

ok so one thing i found when watching the system tra while PS is computing.

processir load 1%
memory slowly going up
network is heavily loaded!!!!

so actually having all the photos loaded in PS doesnt mean they are loaded into the memory and with every step its taking its always reloading them from the network drive which is crazy.

is there a way how to keep all the photos loaded into the memory whn i have enough RAM? 256 gb ATM, but possible to have 1TB if needed...

So in the end it seems that placing the PHOTS to the local drive fixes the problem.

Having option to load the photos into memory once would be really great.

Title: Re: Calculating vertex colors - stuck
Post by: Alexey Pasumansky on July 03, 2017, 01:43:00 PM

Hello zmeski,

Mesh generation process is not utilizing the original photos during the vertex colors calculation, these colors are calculated from the point cloud used as a source for the operation.

Title: Re: Calculating vertex colors - stuck
Post by: zmeski on July 03, 2017, 02:01:38 PM

OK but evidenty something is going wrong there.

When i have the assets saved ont he network, the mesh creation takes forever to the point where its utilizing 1% of the processor and the network activity is going crazy with PS.
When i have the files placed locally, everything is smooth and quick.

Title: Re: Calculating vertex colors - stuck
Post by: Alexey Pasumansky on July 03, 2017, 02:10:26 PM

Hello zmeski,

If you are using project in PSX format, the dense cloud will be accessed via network during the vertex color generation process.

What's the speed of network connection in your case?

Title: Re: Calculating vertex colors - stuck
Post by: zmeski on July 03, 2017, 02:39:32 PM

i think its good old 100Mbit line here :-/