Messages

1

Python and Java API / Unprojecting coordinates (bug?)

« on: July 17, 2024, 11:35:30 AM »

Hi all,

I'm using Agisoft Metashape as part of my workflow studying animal behaviour. I used the software to construct an orthomosaic and subsequently obtained drone videos of animals moving in this area. I then used computer vision algorithms to obtain coordinates of animals in these videos. To then convert these x-y coordinates from pixel space to geographical coordinates, I used the following code:

Code: [Select]

import Metashape
import pandas as pd

# Load the Metashape document and access the first chunk and its surface model
doc = Metashape.app.document
chunk = doc.chunks[0]
surface = chunk.model

# Switch to a different chunk for processing
chunk = doc.chunks[2]
print(f"Processing Chunk: {chunk.label}")

# Read the CSV file into a DataFrame
csv_path = '/Users/vivekhsridhar/Library/Mobile Documents/com~apple~CloudDocs/Documents/Metashape/TalChhapar/output/test_uv.csv'
df = pd.read_csv(csv_path)

# Initialize an empty list to store the data
data = []

# Iterate through cameras in the chunk
for camera in chunk.cameras:

# Filter the DataFrame to only include rows related to a specific camera
df_filtered = df[df['Camera'] == camera.label]

# Iterate through the filtered DataFrame and process each point
for index, row in df_filtered.iterrows():
    idx = row['idx']
    camera_label = row['Camera']
    u = row['u']
    v = row['v']

    # Create a 2D vector from the coordinates
    coords_2D = Metashape.Vector([u, v])
   
    # Pick a point on the model surface using the camera's center and the unprojected 2D coordinates
    point_internal = surface.pickPoint(camera.center, camera.unproject(coords_2D))

    # Transform the internal 3D point to world coordinates
    point3D_world = chunk.crs.project(chunk.transform.matrix.mulp(point_internal))

    # Append the data to the list
    data.append({
        'idx': idx,
        'Camera': camera_label,
        'u': u,
        'v': v,
        'x': point3D_world.x,
        'y': point3D_world.y,
        'z': point3D_world.z
    })

# Convert the list to a DataFrame
df_output = pd.DataFrame(data)

# Save the DataFrame to a CSV file
output_csv_path = '/Users/vivekhsridhar/Library/Mobile Documents/com~apple~CloudDocs/Documents/Metashape/TalChhapar/output/test_3D_world.csv'
df_output.to_csv(output_csv_path, index=False)

print(f"3D world coordinates saved to {output_csv_path}")

I know the code works because I've tested it with data from a single frame. However, when I run entire videos, it fails with the following error message.

2024-07-17 10:21:31 Traceback (most recent call last):
2024-07-17 10:21:31   File "/Users/vivekhsridhar/Library/Mobile Documents/com~apple~CloudDocs/Documents/Metashape/TalChhapar/scripts/test.py", line 40, in <module>
2024-07-17 10:21:31     point3D_world = chunk.crs.project(chunk.transform.matrix.mulp(point_internal))
2024-07-17 10:21:31 TypeError: argument 1 must be Metashape.Vector, not None
2024-07-17 10:21:31 Error: argument 1 must be Metashape.Vector, not None

I understand this occurs because my variable point_internal returns None. However, this shouldn't occur. I manually examined one specific case where the software unprojects the pixel coordinates (5025.5, 1802.5) to None. However, even a small change in the coordinates (5025.50001, 1802.5) or (5025.49999) works well and the software unprojects the coordinate to return reasonable values for the variable point_internal.

Since I'm not looking for that level of precision, I will proceed by jittering the value by small amounts when the software returns None and this will solve my problem. I however wanted to report this in case this is a bug with the software? If it isn't a bug, I would of course also like to understand what's causing it.

Best,
Vivek

2

Python and Java API / Re: Projecting from image coordinates to the orthomosaic

« on: July 09, 2024, 05:39:31 PM »

Sorry. Problem resolved. The error came from the fact that the altitude values on my image EXIFs were wrong. So the entire result was just a rescaled version of what it needed to be.

Best,
Vivek

3

Python and Java API / Re: Projecting from image coordinates to the orthomosaic

« on: July 09, 2024, 04:24:12 PM »

Hi all,

I'm still struggling with this. I'd really appreciate if someone could help me solve this problem and so, I'm recapping what I've done so far here.

I'm a biologist that studies animal movement and behaviour. As part of my work, I acquired UAV images which I used in Agisoft Metashape to reconstruct a georeferenced map (an orthomosaic) of my animals' habitat. As a next step, I have videos I collected within this habitat. I have used computer vision algorithms to track my animals within my video, thus giving me their location in pixel coordinates. I would now like to convert trajectories of these animals from pixel space to global coordinates. To test if this would work, I first marked specific points on my orthomosaic. I then converted these coordinates to image space (from 3D to 2D) using the following code (note the images are part of a separate chunk which I've aligned separately. I've also georeferenced this chunk using GCPs and the tiepoints align beautifully to the original chunk where I created the orthomosaic):

Code: [Select]

import Metashape
import numpy as np
import pandas as pd

def list_coordinates(file_path):
    data = []
    with open(file_path, 'r') as file:
        for line in file:
            values = line.strip().split(',')[1:-1]
            # Filter out empty strings before converting to float
            values = [float(val) if val else 0.0 for val in values]
            data.append(tuple(values))
    return data

def process_chunk(chunk, global_coordinates):
    T = chunk.transform.matrix

    data_list = []

    for point_idx, global_coord in enumerate(global_coordinates):
        p = T.inv().mulp(chunk.crs.unproject(Metashape.Vector(global_coord)))
        print(f"Image pixel coordinates of point {point_idx + 1} with global coordinates ({chunk.crs.name}): {global_coord}")

        for i, camera in enumerate(chunk.cameras):
            project_point = camera.project(p)

            if project_point:
                u = project_point.x  # u pixel coordinates in camera
                v = project_point.y  # v pixel coordinates in camera

                if 0 <= u <= camera.sensor.width and 0 <= v <= camera.sensor.height:
                    # Extract video and frame_seq from the camera label
                    camera_label_parts = camera.label.split('_')
                    video = '_'.join(camera_label_parts[:-1])
                    frame_seq = camera_label_parts[-1]

                    data_list.append([point_idx + 1, camera.label, video, frame_seq, u, v])

    columns = ['Point', 'Camera', 'video', 'frame_seq', 'u', 'v']
    df = pd.DataFrame(data_list, columns=columns)
    return df

# Define global coordinates for multiple points
points = list_coordinates('/Users/vivekhsridhar/Library/Mobile Documents/com~apple~CloudDocs/Documents/Metashape/TalChhapar/output/ninety_territory_points_xyz.txt')

# Use active Metashape document
doc = Metashape.app.document

# Iterate through all chunks in the document
df = pd.DataFrame()
for chunk in doc.chunks:
    if chunk.label != 'Chunk 1' and chunk.label != 'Chunk 2':
        # Set the current chunk to the one being processed
        doc.chunk = chunk

        # Process the current chunk
        tmp = process_chunk(chunk, points)
        df = pd.concat([df, tmp], ignore_index=True)

# Save the results to a CSV file
df.to_csv('/Users/vivekhsridhar/Library/Mobile Documents/com~apple~CloudDocs/Documents/Metashape/TalChhapar/output/p1_territory_points_uv.csv', index=False)

I know this code works because I plotted the 2D coordinates onto the respective images and they appear at the right location on the image (attached sample image). So next, I decided to convert these 2D coordinates back to 3D and check if I obtained coordinates of my original points. I used the following code to do this:

Code: [Select]

import Metashape
import datetime
import pandas as pd

doc = Metashape.app.document
chunk = doc.chunks[0]
surface = chunk.point_cloud

chunk = doc.chunks[2]
print(chunk.label)

camera = chunk.cameras[0]
print(camera)

df = pd.read_csv('/Users/vivekhsridhar/Library/Mobile Documents/com~apple~CloudDocs/Documents/Metashape/TalChhapar/output/p1_territory_points_uv.csv')
df = df[df['Camera'] == '20230310_SE_Lek1_P1D1_DJI_0190_0000']

for index, row in df.iterrows():
x = row['u']
y = row['v']

coords_2D = Metashape.Vector([x, y])
point_internal = surface.pickPoint(camera.center, camera.unproject(coords_2D))
point3D_world = chunk.crs.project(chunk.transform.matrix.mulp(point_internal))

print(point_internal)
print(point3D_world)

This however does not work and the newly obtained 3D coordinates do not match the location of the original point on the orthomosaic. Where have I gone wrong? Unlike my previous post, all my chunks now have an [R] flag next to them so all images are referenced as well. I'd appreciate any help in terms of how I can proceed with this.

Best,
Vivek

4

General / Aligning photos across years

« on: July 01, 2024, 10:45:20 AM »

Dear Metashape Community,

I'm an animal behaviour researcher using metashape to reconstruct the landscape where the animals move. For this, I'm looking to create orthomosaics of the area. As part of the process, I've collected UAV-based images of the same location two years in a row (2023 and 2024). Unfortunately in 2023, I did not use markers or gather images systematically using a flight plan. Hence, the error on this map is much higher than the error I have on the 2024 map (where I used 5 markers and used an automated flight plan to gather images with 75% overlap).

Since I have good animal movement data from 2023, I would like to align the 2023 map to the 2024 map to reduce error. While the landscape has changed considerably in parts, there are roads and tracks that can be used to align one map to the other. Does the following pipeline make sense to do this?

1. Align photos 2023 in chunk 1
2. Align photos 2024 in chunk 2
3. Align chunks with 2024 as reference
4. Create DEM 2023 followed by the orthomosaic

I was wondering if this workflow makes sense. Since all my images are geotagged, both chunks 1 and 2 have [R] next to them. Does this matter when I use the align chunks function? I was curious as in most posts I find, users align chunks and then merge them. This seems like a way to break the computational requirement. In my case however, I would like both these chunks to be kept separate as they represent data from separate years and only align chunks to improve accuracy of older data.

Thank you for your help!

Best,
Vivek

5

Python and Java API / Re: Projecting from image coordinates to the orthomosaic

« on: January 19, 2024, 10:57:18 AM »

Hi again,

I was wondering if someone could confirm that the lack of image calibration could in fact result in this issue? And if so, is there a way in metashape to import frames from a video while also keeping the intrinsic parameters for calibration? Alternatively, since it is the same camera / drone that was used to create the orthomosaic, would it be a possibility to use the same calibration parameters?

Thanks again for helping me with this.

Vivek

6

Python and Java API / Re: Projecting from image coordinates to the orthomosaic

« on: January 18, 2024, 02:36:09 PM »

I might have found the issue. In the code where I convert the U,V image coordinates back to the X,Y,Z orthomosaic, the unprojection seems to work if I apply it to photos within the same chunk as the orthomosaic. However, this does not seem to work when I do the unprojection on images on a separate chunk. For example, in the code below, everything seems to work once I silence the line "chunk = doc.chunks[1]" (as done below).

Code: [Select]

import Metashape
import datetime


doc = Metashape.app.document
chunk = doc.chunk
surface = chunk.point_cloud

# chunk = doc.chunks[1]
print(chunk.label)

camera = chunk.cameras[0]
print(camera)

x=2066
y=544

coords_2D = Metashape.Vector([x, y])
point_internal = surface.pickPoint(camera.center, camera.unproject(coords_2D))
point3D_world = chunk.crs.project(chunk.transform.matrix.mulp(point_internal))

print(point_internal)
print(point3D_world)

I checked and the other chunk (which I'm trying to project onto the orthomosaic) is aligned to the base chunk. However, the photos have aren't calibrated (they have an NC next to them). Could this be the issue? Here is the code I use to align images on the new chunk with my 'BaseMap'.

Code: [Select]

import Metashape
import datetime
import glob
import os
import pathlib
import numpy as np
import helper_functions as hf

doc = Metashape.app.document
reference_chunk = doc.chunk.key

# The directory where videos are saved
VideoDirectory = "/Users/vivekhsridhar/Library/Mobile Documents/com~apple~CloudDocs/Documents/Metashape/Solitude/images/"

# The directory where frames must be imported
ImportDirectory = "/Users/vivekhsridhar/Library/Mobile Documents/com~apple~CloudDocs/Documents/Metashape/Solitude/images/"

def list_videos(directory):
    directory_path = pathlib.Path(directory)
    video_extensions = ['.mp4', '.avi', '.mkv', '.mov', '.wmv']
    video_files = [(str(file.resolve()), file.stem) for file in directory_path.iterdir() if file.suffix.lower() in video_extensions]
    return video_files
def import_frames(video_path, video_name, import_directory):
    # Create a folder for each video
    video_folder = os.path.join(import_directory, video_name)
    os.makedirs(video_folder, exist_ok=True)

    # Add a new chunk for each video
    chunk = doc.addChunk()
    chunk.label = video_name
    doc.chunks.append(chunk)

    # Define the image names pattern for frames
    image_names = os.path.join(video_folder, f'{video_name}_{{filenum:04}}.png')

    # Import frames with custom frame step
    chunk.importVideo(video_path, image_names, frame_step=Metashape.FrameStep.CustomFrameStep, custom_frame_step=20)


hf.log( "--- Starting workflow ---" )
hf.log( "Metashape version " + Metashape.Application().version )
hf.log_time()

# List videos in VideoDirectory
videos = list_videos(VideoDirectory)

# Import frames from listed videos
for video_path, video_name in videos:
    import_frames(video_path, video_name, ImportDirectory)


# Parameters for feature matching photos
match_photos_config = {
    'downscale': 1,
    'generic_preselection': True,
    'reference_preselection': True,
    'reference_preselection_mode': Metashape.ReferencePreselectionEstimated
}

chunk_dirs = hf.get_subdirectories(ImportDirectory, ignore_files=['.DS_Store'])

all_chunks = [reference_chunk]
# Match photos and align cameras
for chunk in doc.chunks:
    if chunk.label != 'BaseMap':
        hf.log( "Processing chunk" )

        chunk.matchPhotos(**match_photos_config)
        chunk.alignCameras()
        all_chunks.append(chunk.key)

doc.alignChunks(chunks=all_chunks, reference=reference_chunk)

hf.log_time()
hf.log( "--- Finished workflow ---")

doc.save('/Users/vivekhsridhar/Library/Mobile Documents/com~apple~CloudDocs/Documents/Metashape/Solitude/Solitude.psx')

How would I go about fixing this? I've also attached a screenshot of my workspace if that helps.

Cheers!
Vivek

7

Python and Java API / Re: Projecting from image coordinates to the orthomosaic

« on: January 18, 2024, 02:12:56 PM »

I used a DEM surface model. I haven't classified my ground points

8

Python and Java API / Re: Projecting from image coordinates to the orthomosaic

« on: January 18, 2024, 02:00:01 PM »

Hi Paul, thanks for taking the time to try and sort this out. I'm using version 2.0.4. So this should still be the dense cloud then?

9

Python and Java API / Re: Projecting from image coordinates to the orthomosaic

« on: January 18, 2024, 01:00:55 PM »

All coordinate systems seem to match.

I wrote these two lines of code to output crs of the orthomosaic and the shapes

Code: [Select]

import Metashape

doc = Metashape.app.document
chunk = doc.chunk

print('crs of orthomosiac is: ', chunk.orthomosaic.crs)
print('crs of shapes is: ', chunk.shapes.crs)

This is the output on the Metashape console
2024-01-18 10:53:19 crs of orthomosiac is:  <CoordinateSystem 'WGS 84 (EPSG::4326)'>
2024-01-18 10:53:19 crs of shapes is:  <CoordinateSystem 'WGS 84 (EPSG::4326)'>

As suggested by you, I also checked this on the chunk info (images attached).

I also checked the settings on the reference pane for the crs of the photos and the markers and they all also seem to be WGS 84 (EPSG::4326) (Screenshot attached).

10

Python and Java API / Re: Projecting from image coordinates to the orthomosaic

« on: January 18, 2024, 12:39:57 PM »

The initial global coordinates were extracted from the orthomosaic. Here, I use the 'draw point' function to click on specific features on the orthomosaic. This created a shape layer with my marked points. I exported this shape layer as a .txt (file attached). These are the X,Y and Z coordinates that I start with. The subsequent processing is as described before. I use my first script to convert these X,Y,Z coordinates to U,V image coordinates and then try to obtain the X,Y,Z back by unprojecting the U,V coordinates.

11

Python and Java API / Re: Projecting from image coordinates to the orthomosaic

« on: January 18, 2024, 11:56:07 AM »

@Paul, thanks for the quick response. I just checked and the chunk.crs in both scripts are the same (CoordinateSystem 'WGS 84 (EPSG::4326)'). Let me know if you spot some other bug in my code / error in my thought process.

Cheers!
Vivek

12

Python and Java API / Projecting from image coordinates to the orthomosaic

« on: January 17, 2024, 06:00:15 PM »

Hi all,

I'm a biologist that studies animal movement and behaviour. As part of my work, I acquired UAV images which I used in Agisoft Metashape to reconstruct a georeferenced map of my animals' habitat. As a next step, I have videos I collected within this habitat. I have used computer vision algorithms to track my animals within my video, thus giving me their location in pixel coordinates. I would now like to convert trajectories of these animals from pixel space to global coordinates. To test if this worked, I first marked specific points on my orthomosaic. I then converted these coordinates to image space (from 3D to 2D) using the following code:

Code: [Select]

import numpy as np
import pandas as pd
import Metashape

def list_coordinates(file_path):
    data = []
    with open(file_path, 'r') as file:
        for line in file:
            values = line.strip().split(',')[1:-1]
            # Filter out empty strings before converting to float
            values = [float(val) if val else 0.0 for val in values]
            data.append(tuple(values))
    return data

def process_chunk(chunk, global_coordinates):
    T = chunk.transform.matrix

    data_list = []

    for point_idx, global_coord in enumerate(global_coordinates):
        p = T.inv().mulp(chunk.crs.unproject(Metashape.Vector(global_coord)))
        print(f"Image pixel coordinates of point {point_idx + 1} with global coordinates ({chunk.crs.name}): {global_coord}")

        for i, camera in enumerate(chunk.cameras):
            project_point = camera.project(p)

            if project_point:
                u = project_point.x  # u pixel coordinates in camera
                v = project_point.y  # v pixel coordinates in camera

                if 0 <= u <= camera.sensor.width and 0 <= v <= camera.sensor.height:
                    # Extract video and frame_seq from the camera label
                    camera_label_parts = camera.label.split('_')
                    video = '_'.join(camera_label_parts[:-1])
                    frame_seq = camera_label_parts[-1]

                    data_list.append([point_idx + 1, camera.label, video, frame_seq, u, v])

    columns = ['Point', 'Camera', 'video', 'frame_seq', 'u', 'v']
    df = pd.DataFrame(data_list, columns=columns)
    return df

# Define global coordinates for multiple points
points = list_coordinates('/Users/vivekhsridhar/Library/Mobile Documents/com~apple~CloudDocs/Documents/Metashape/Solitude/output/points_xyz.txt')

# Use active Metashape document
doc = Metashape.app.document

# Iterate through all chunks in the document
df = pd.DataFrame()
for chunk in doc.chunks:
    if chunk.label != 'BaseMap':
        # Set the current chunk to the one being processed
        doc.chunk = chunk

        # Process the current chunk
        tmp = process_chunk(chunk, points)
        df = pd.concat([df, tmp], ignore_index=True)

# Save the results to a CSV file
df.to_csv('/Users/vivekhsridhar/Library/Mobile Documents/com~apple~CloudDocs/Documents/Metashape/Solitude/output/points_uv.csv', index=False)

I know this code works because I plotted the 2D coordinates onto the respective images and they appear at the right location on the image. So next, I decided to convert these 2D coordinates back to 3D and check if I obtained coordinates of my original points. I used the following code to do this:

Code: [Select]

import Metashape
import datetime


doc = Metashape.app.document
chunk = doc.chunk
surface = chunk.point_cloud

chunk = doc.chunks[1]
print(chunk.label)

camera = chunk.cameras[0]
print(camera)

x=337.67729912777475
y=903.5153164982171

coords_2D = Metashape.Vector([x, y])
point_internal = surface.pickPoint(camera.center, camera.unproject(coords_2D))
point3D_world = chunk.crs.project(chunk.transform.matrix.mulp(point_internal))

print(point_internal)
print(point3D_world)

This however does not work and the newly obtained 3D coordinates do not match the location of the original point on the orthomosaic. Where have I gone wrong?

Thanks in advance for your help! I really appreciate any pointers in this direction.

Cheers!
Vivek