Build wheels on Ubuntu CI takes a long time

### Description

This one is taking over 40 minutes already, making it the longest job in the CI: https://github.com/pymc-devs/pytensor/actions/runs/4172705299/jobs/7224173170

Can we run it only in release?