Hello GeoGecco,
I think you've got the DEM (or mesh) surface that is extrapolated around the actual object, so PhotoScan is trying to project the images on that surface.
If the selected (assigned) image doesn't contain the information for the whole area, such missing gaps will be filled from other cameras.
So basically speaking, the assigned image is properly applied to the area, but for the sub-sections of the selected polygon where the assigned image doesn't have any information, PhotoScan has applied other images.
I think you need to get the surface without the additional extrapolated geometry to avoid this effect.