I don't think this is fair what my manager is doing. Can you help?

Be happy. If something turns up missing off hours they can't blame you. You don't have a key. Look for the bright side.