The reference tool is “locked” with the canvas, as in, it will zoom when the canvas is zoomed, and it will mirror when the canvas is mirrored… from what I know it’s normal, so you’re not doing anything wrong.
If it really bothers you, @freyalupen created a plugin that you can download:
It puts the reference in its own window so you can zoom in on your drawing/painting without losing the reference from sight. I tested this one and it’s working pretty well!