Hello Jan,
If the format is not not specified, then PhotoScan will parse the export path and use the extension to identify the format to be used. In case the format is specified but doesn't match the extension, PhotoScan will export the data according to the format use (PointsFormat), but save it under the user-defined path, without changing anything in it.
In the current 1.3 pre-release TXT extension is not interpreted as XYZ output, so for such extension you need to specify PointsFormatXYZ explicitly.
If you can reproduce the broken or incomplete export for XYZ format, please let me know.