You’re right—that code should work. Two likely culprits:
Hook order — something else (plugin/theme) is resetting posts_per_page after your handler.
Detecting the taxonomy can be flaky; grabbing the queried term is more reliable.
Try this hardened version: runs late (priority 100) and inspects the queried object.