Please improve the algorithm for building texture with blocked model.

[steve3d]

I have a large dataset, which contains about 33k photos, Because there is no way to edit the mesh of tiled model, So I decided to build as blocked model.

After build with blocked, model, this dataset created 299 blocks, with settings of block size=250m. So I started to create texture for this blocked model, but the texture process takes extreme long than I expected.

For each textureing process of a block, it first need to read all 299 blocks, this step need about 2300-2500 seconds, and every block need this step, here is part of one block process log:

Code: [Select]

2024-04-07 16:48:37 loaded mesh in 4.142 sec
2024-04-07 16:48:38 loaded mesh in 0.557 sec
2024-04-07 16:48:38 loaded mesh in 0.327 sec
2024-04-07 16:48:38 done in 2342.6 sec
2024-04-07 16:48:38 Blending textures...
2024-04-07 16:48:38 Done "load_model" in 0

but this block's total process time is 3700 seconds:

Code: [Select]

2024-04-07 17:00:24 subdividing model... done in 0 sec
2024-04-07 17:00:24 extracting tiles... done in 0 sec
2024-04-07 17:00:24 parameterizing tiles... done in 151.889 sec
2024-04-07 17:03:43 writing 2920 tiles... done in 0.519 sec
2024-04-07 17:03:44 writing 717 tiles... done in 0.089 sec
2024-04-07 17:03:44 writing 128 tiles... done in 0.017 sec
2024-04-07 17:03:44 writing 26 tiles... done in 0.001 sec
2024-04-07 17:03:44 writing 4 tiles... done in 0 sec
2024-04-07 17:03:44 writing 1 tiles... done in 0 sec
2024-04-07 17:03:51 processing finished in 3791.55 sec

so this means for building texture of this large dataset, I need minimum of 299 hours, and 63% of this time are wasted by every blocks repeating reading of each block.

This takes way too much time of wasting. So, is there any way of  improve the algorithm of building texture for blocked model?

[steve3d]

This might not be a problem for small datasets, some small datasets only need few blocks, but for large, even city wide dataset, this is really a serious problem.

I even noticed some small block, the real processing time only need about 60-600 seconds, but reading all blocks still need 2300-2600 seconds. This means for these kind of blocks, over 80% of time are reading blocks, from my perspective, this is not acceptable.

[Alexey Pasumansky]

Hello steve3d,

The procedure of block model texturing will be improved in 2.1.2.

[steve3d]

Hello Pasumansky,  above case finally failed because there is not enough memory to finally parameterize the blocks in generating texture step. I'm using a computer which has 128G memory, this failed eventually.

Any chance to improve this memory usage in such case?

[steve3d]

and another thing, is it possible to make these steps parallel?

Code: [Select]

2024-04-11 09:27:18 saved group #259/384: 182.68 MB cubes, 259.858 MB/s, 39.0955 compressed MB - i.e. 21% compression
2024-04-11 09:27:18 2 cameras done in 13.596 s
2024-04-11 09:27:18 loading 2 cameras...
2024-04-11 09:27:19 generating cubes...
2024-04-11 09:27:30 total: 30438815 samples, 12176217 image cubes, 12.8872 avg level
2024-04-11 09:27:31 saving 10985790 merged group cubes ~36%...
2024-04-11 09:27:31 saved group #260/384: 167.63 MB cubes, 261.106 MB/s, 34.4178 compressed MB - i.e. 21% compression
2024-04-11 09:27:31 2 cameras done in 12.859 s
2024-04-11 09:27:31 loading 2 cameras...
2024-04-11 09:27:32 generating cubes...
2024-04-11 09:27:40 total: 13249102 samples, 5170086 image cubes, 12.6206 avg level
2024-04-11 09:27:40 saving 4776529 merged group cubes ~36%...
2024-04-11 09:27:41 saved group #261/384: 72.884 MB cubes, 114.598 MB/s, 14.7257 compressed MB - i.e. 20% compression
2024-04-11 09:27:41 2 cameras done in 9.416 s
2024-04-11 09:27:41 loading 2 cameras...
2024-04-11 09:27:42 generating cubes...
2024-04-11 09:27:49 total: 12203757 samples, 5281054 image cubes, 13.1447 avg level
2024-04-11 09:27:49 saving 4356566 merged group cubes ~36%...
2024-04-11 09:27:49 saved group #262/384: 66.4759 MB cubes, 226.109 MB/s, 15.0156 compressed MB - i.e. 23% compression
2024-04-11 09:27:49 2 cameras done in 8.577 s
2024-04-11 09:27:49 loading 2 cameras...
2024-04-11 09:27:50 generating cubes...
2024-04-11 09:27:59 total: 23498082 samples, 9017552 image cubes, 12.5799 avg level
2024-04-11 09:28:00 saving 7712995 merged group cubes ~33%...
2024-04-11 09:28:01 saved group #263/384: 117.691 MB cubes, 248.818 MB/s, 24.9984 compressed MB - i.e. 21% compression
2024-04-11 09:28:01 2 cameras done in 11.161 s
2024-04-11 09:28:01 loading 2 cameras...
2024-04-11 09:28:02 generating cubes...
2024-04-11 09:28:12 total: 19114522 samples, 7384658 image cubes, 12.678 avg level
2024-04-11 09:28:12 saving 7384658 merged group cubes ~39%...
2024-04-11 09:28:13 saved group #264/384: 112.681 MB cubes, 148.264 MB/s, 21.9049 compressed MB - i.e. 19% compression
2024-04-11 09:28:13 2 cameras done in 12.321 s
2024-04-11 09:28:13 loading 2 cameras...
2024-04-11 09:28:14 generating cubes...
2024-04-11 09:28:26 total: 30670067 samples, 12238388 image cubes, 12.6715 avg level
2024-04-11 09:28:27 saving 11891791 merged group cubes ~39%...

each step only process 2 camera, and each step only use one core to process. when building large models, this also waste too much time.

[Alexey Pasumansky]

Hello steve3d,

Reworking texturing procedure in general is already in progress, but it would like take quite a while to be implemented in the release version, so currently in order to reduce the memory consumption for the block model texture generation you need either to use lower texture resolution (so smaller number of pages per block are generated) or to reduce the size of initial model blocks.

For a very rough estimation for the number of texture pages and RAM consumption you can use the following code, but adjust the input values: ghosting filter enabled/disabled, block size, texture page size, output resolution:

Code: [Select]

K = 3 #surface complexity and atlas filling coefficient
K_ghost = 60 #60 - with ghosting filter, 36 - without ghosting filter
texture_size = 16384 #pixels
block_size = 25 #meters
resolution = 0.00075 #m/pix resolution
N_pages = int((block_size / resolution / texture_size) ** 2 * K) + 1
req_memory = texture_size ** 2 * K_ghost * N_pages
print(N_pages, "texture pages, ", req_memory / 1024 ** 3, "GB")


[steve3d]

thank you for the detail explanation, according to the codes above, If I change the block size from 250 to 450, this will reduce the number of blocks from 300 to about 80-90. without reducing the pixel size of texture, this should be able to generate the texture on a 128G memory.

I'll try again. but the model at this scale, takes very long time to generate.

And another suggestion, when building the texture for blocked model, I think it's better to give a memory requirement estimation in the dialog.