Forum

Author Topic: import csv image in photoscan  (Read 7141 times)

tuffi

  • Jr. Member
  • **
  • Posts: 67
    • View Profile
import csv image in photoscan
« on: June 09, 2016, 04:29:47 PM »
Dear friends,
I need help for analyze some thermal image dataset.
I have some thousand of .CSV matrix that are a 382x288 array, where every cells are a temperature in celsius with one or two decimal data.
I need to make a photogrammetric process to my data, so i need to convert my CSV data into TIFF image (16bit - the sensor works at 12bit). So:
1 - Looking for a minimum and maximum values of temperature of all dataset of array
2 - Create a LUT where the min temp is black (0) and max temp is white (65,536)
3 - Whit this palette create a TIFF image of every CSV.
After photogrammetric recostruction I need to resample final orthoimage with the same LUT and create a 32 bit TIFF possible with radiometric information for every pixel.
Anyone know some script that can help me!?
Thank you so much..

tuffi

  • Jr. Member
  • **
  • Posts: 67
    • View Profile
Re: import csv image in photoscan
« Reply #1 on: June 10, 2016, 01:05:45 PM »
...of course the perfect way is make it direct from phyton console and import the created TIFF directly in Photoscan

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 13729
    • View Profile
Re: import csv image in photoscan
« Reply #2 on: June 10, 2016, 02:07:56 PM »
Hello tuffi,

I think you can create PhotoScan.Image() instance of a required resolution and fill it's pixel data with the information from CSV, then save the images to disk and load them as photos.
Best regards,
Alexey Pasumansky,
Agisoft LLC

tuffi

  • Jr. Member
  • **
  • Posts: 67
    • View Profile
Re: import csv image in photoscan
« Reply #3 on: June 10, 2016, 03:24:32 PM »
Thank you,

so, for the first stemp (1 - Looking for a minimum and maximum values of temperature of all dataset of array) I need to work outside Photoscan, right?


I acquire data from OPTRIS PI400 camera...

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 13729
    • View Profile
Re: import csv image in photoscan
« Reply #4 on: June 16, 2016, 09:39:47 PM »
Hello tuffi,

I think that the first step can be also incorporated to the script. If I have understood the input format correctly, you can loop through all the input csv files to get min and max values, then convert the csv to PhotoScan.Image() taking into account the values thresholds.

Can you provide a couple of sample files in this format and also specify, where the images should be normalized accross the whole image set or for each image individually?
Best regards,
Alexey Pasumansky,
Agisoft LLC

tuffi

  • Jr. Member
  • **
  • Posts: 67
    • View Profile
Re: import csv image in photoscan
« Reply #5 on: June 17, 2016, 12:47:08 PM »
Yes, of course.

I attach the files.

Thanks
E.t.

tuffi

  • Jr. Member
  • **
  • Posts: 67
    • View Profile
Re: import csv image in photoscan
« Reply #6 on: June 17, 2016, 05:58:26 PM »
The normalization is need on all set of image.....

So, if black (RGB: 0.0.0) is 20 degrees in one image, must be 20 degrees in all image set..(if in one file there is no 20 degree data..so in this image the value RGB 0,0,0 will be not present)...

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 13729
    • View Profile
Re: import csv image in photoscan
« Reply #7 on: June 19, 2016, 08:52:24 PM »
Hello tuffi,

Thank you for providing the sample data, we'll try to check what could be done with that.
Best regards,
Alexey Pasumansky,
Agisoft LLC

tuffi

  • Jr. Member
  • **
  • Posts: 67
    • View Profile
Re: import csv image in photoscan
« Reply #8 on: June 22, 2016, 12:35:56 PM »
Thank you so much Alexey!

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 13729
    • View Profile
Re: import csv image in photoscan
« Reply #9 on: March 14, 2017, 12:38:59 PM »
Hello tuffi,

Maybe it's already too late, but here's the code that reads the data from the csv files in the user-defined folder, scales the data to 0 - 65535 range using min and max values across the dataset, creates 16 bit TIFF images and loads them back to PhotoScan:

Code: [Select]
#compatibility PhotoScan Professional 1.3.0

import PhotoScan, os

main_path = PhotoScan.app.getExistingDirectory("Specify the folder with the CSV files:")

WIDTH = 382 #640
HEIGHT = 288 #512
DELIMITER = ";" #","
ABS_MAX = 65535
ABS_MIN = 0

doc = PhotoScan.app.document
chunk = doc.addChunk()
chunk.label = "Chunk thermal"

files = os.listdir(main_path)
thermal = list()
min_t, max_t = 10.0E+10, 10.0E-10

#finding min and max
for file in files:

if file[-3:].lower() != "csv":
continue
file_csv = open(main_path + "/" + file, "rt")
for y in range(HEIGHT):
line = file_csv.readline()
pixels = [float(x) for x in line.strip().split(DELIMITER) if x]
max_t = max(max(pixels), max_t)
min_t = min(min(pixels), min_t)
file_csv.close()

#creating images
for file in files:

if file[-3:].lower() != "csv":
continue

file_csv = open(main_path + "/" + file, "rt")

image = PhotoScan.Image(WIDTH, HEIGHT, " ", "U16")

for y in range(HEIGHT):
line = file_csv.readline()
pixels = [float(x) for x in line.strip().split(DELIMITER) if x]
for x in range(WIDTH):
pixel = pixels[x] * (ABS_MAX - ABS_MIN) / (max_t - min_t)
image[x,y] = (pixel,)
file_csv.close()

image.save(main_path + "/" + file[:-3] + "tif")
thermal.append(main_path + "/" + file[:-3] + "tif")

doc.chunk = chunk
chunk.addPhotos(thermal)

print("script finished")
Best regards,
Alexey Pasumansky,
Agisoft LLC

gio_scan

  • Newbie
  • *
  • Posts: 3
    • View Profile
Re: import csv image in photoscan
« Reply #10 on: January 03, 2018, 05:57:19 PM »
Hi there; new to the forum and this is my first post; first of all Happy 2018!!!
I know this thread is old, but I'll try anyways :)
@ Alexey - your script to convert the Optris CSVs' into linear 16 bit TIFFs is intended to process one batch/chunk.
In case of large area surveys/missions, multiple flights are needed requiring one or more days to cover the whole target area; this means there will be several batches/chunks, each with its own set of CSVs, and individual max and min temperatures (which will also be influenced by the ambient temperature that obviously changes throughout the day). As an example with round numbers, let's assume that the target mission is to survey a 15 ha area: this is accomplished with 10 flights, and each flight generates 400 thermal images (in CSV format).
To generate the whole orthophoto/3D model of the entire 15 ha area, all the 4000 CSV files could be processed at once: while this will correctly take into account the absolute max and min temperatures of the entire 15 ha area recorded during the 10 flights, and will generate the re-scaled linear 16 bit TIFF images to be processed, I think it can be very hardware intensive and time-taking.
Now my question: is thee a way to process each of the 10 batches/chunks individually, and then mosaic them taking into account the individual max and min temperatures, in order to get the overall "temperature normalized" orthophoto?
Thanks a lot
Regards
Giovanni

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 13729
    • View Profile
Re: import csv image in photoscan
« Reply #11 on: January 03, 2018, 07:32:15 PM »
Hello Giovanni,

As far as I remember, the script creates the new chunk, then generated 16-bit TIFFs from the CSV files and loads them to the chunk as images. So in principle, you can run the script several times and create multiple chunks, then merge the chunks and process them in the single block.
Alternatively, you can process them individually and either export the orthomosaic from each block.
Best regards,
Alexey Pasumansky,
Agisoft LLC

gio_scan

  • Newbie
  • *
  • Posts: 3
    • View Profile
Re: import csv image in photoscan
« Reply #12 on: January 03, 2018, 07:55:26 PM »
Thanks a lot Alexey for the prompt reply.
So, either processing methods you suggested will generate a final orthophoto (from the individual chunks) which takes into account the different "scale" (DNs linear with temperatures from CSVs) of each individual chunks?
Thanks
Giovanni

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 13729
    • View Profile
Re: import csv image in photoscan
« Reply #13 on: January 03, 2018, 07:59:11 PM »
Hello Giovanni,

Yes, if the images are loaded in "chunks" then they will be only scaled according to the actual block.

If you need to normalize all the images at once, then you need to load all the files at once.
Best regards,
Alexey Pasumansky,
Agisoft LLC

gio_scan

  • Newbie
  • *
  • Posts: 3
    • View Profile
Re: import csv image in photoscan
« Reply #14 on: January 04, 2018, 11:47:35 PM »
Hi Alexey; sorry about the delayed reply: I'm just familiarizing myself with Photoscan and the various workflows (I've been using different pieces of SfM and photogrammetry software for my projects; now a potential customer asked me to quote thermal surveys and post-processing with their  hardware/software, and for photogrammetry/3D reconstruction they use Photoscan).
So, if I get it right, the best procedure would be:
- create a "thermal chunk" comprising all the CSV files of a site (multiple flights if that's the case);
- align/optimize all the images;
- duplicate the chunk multiple times (good rule of thumb: take the number of photos, divide by 250, and then duplicate the chunk that number of times).
• Use the Resize Region tool to resize the bounding box in each duplicate chunk to 1/# chunks in size. There should be some overlap between the edges of the bounding boxes to help align the chunks later.
• Then process each chunk separately for the dense cloud and the mesh steps.
• Align the chunks. Merge the chunks.
• Export the DEM/Orthomosaic.
- The "Split in Chunks" script might streamline the process (haven't tried it yet).
I still have a couple of questions to achieve as a detailed and radiometric a process as possible:
- Photo alignment step: I would think "pair selection - generic" should be set; would instead "disable" help?
- Would "blending - disabled" be better than "average"?
Lastly, the 16 bit TIFF derived from the CSVs do not contain any camera/lens parameters: what would be a way to calibrate such images (keeping also in mind that any thermal infrared camera is extremely hard to be optically calibrated).
Thanks for the great help so far: should you have any further tips on this matter I surely would not get offended ;-)
Best Regards
Giovanni
« Last Edit: January 07, 2018, 06:14:21 PM by gio_scan »