Hello indesinenter,
Regarding the first step - Matching Points, do you have "use CPU" flag enabled in the GPU preferences tab? If it is disabled, then CPU is just used to control and GPU processing and perform some minor intermediate steps, while the main part is performed on GPU.
As for the camera locations estimation, do you observe the same behavior throughout the stage or only in the very beginning (before even the 1% of processing has passed)?