Is there any way to remove this permission from Contributors but sill allow members of Contributors + Release Manager + Project Managers to deploy?
I could reproduce the scene you mentioned. It seems that the permissions of the Contributors group will override the permissions of the other two groups.
To solve this issue, you could set the Manage Deployments as Not Set instead of Deny. At the same time, you need the disable Inheritance option.
Note: If you set Not Set, people in the contributor group cannot deploy, but if some of them are in other groups with this permission, they will inherit the permissions from other groups
CLICK HERE to find out more related problems solutions.