How to deselect specific selection?

I have 2 selected drawings and selected a third one to move a part of a drawing.
After I finished moving the drawing, I deselected it but the first 2 deselected as well.
How do I only deselect the third one?

Instead of deselecting, open Tool Options and change the action to subtract. Make a selection around the objects that you no longer want to be selected.