Hello Hamish,
When the mouse cursor is moved while the left mouse button is pressed, the selection rectangle will appear in the pane, so all the previous selection will be discarded.
So it looks like you are too fast moving and clicking the cursor, if you left click with the static cursor, there shouldn't be any "spontaneous" diselections.