Hi engelbeck,
You should NOT reduce the max number of key points to match the number of manual markers. If you reduce this number to your chosen values, it will be impossible for PhotoScan to match and align the images because it has only 1, 5, 10 or 20 points to compare in each image (which is insufficient).
Note that key points are the points that PhotoScan's Structure-from-Motion (SfM) algorithm is automatically extracting from all the images. The best quality key points (= tie points) will subsequently be used to form the sparse point cloud.
You need to keep the key point limit at 40,000 and the tie point limit at 4,000 (default values).
The manually placed markers are an additional source of information that PhotoScan will use. Keep in mind that even if you use manual markers, you need an adequate amount of image overlap to get a good 3D reconstruction.
Also make sure to have CONSTRAIN FEATURE BY MASK ticked in WORKFLOW > ALIGN PHOTOS. If uncheckd, your masked background will be used as well (which could be an additional reason for poor alignment results).
I hope that helps.
Regards,
SAV
P.S. If you still have problems, then please share some screenshots and/or your processing report.