CI: Improve CircleCI

[h-vetinari]

Not sure how much of that needs to be done upstream, but right now, CircleCI is performing very poorly.

We're on the sixth page of jobs (so about 24h waiting time for commits coming in now), and CircleCI is stalling intermittently, e.g. right now, since about half an hour (not sure if those workers may be changing to some private pipelines that are not visible to me):

Further improvements that would speed things up:

• skip jobs also if there are newer force-pushed (e.g. squashed) commits
• respect [ci skip] if issued
• sometimes, only 1-2 workers are running (on change-over between PRs), instead of 4
• more workers?
• etc...

[jreback]

I we somehow got bumped to 1 container, rather than the 4 that we had before. So this is why things are taking forever. I emailed them, let's see.

cc @TomAugspurger

[h-vetinari]

We somehow got bumped to 1 container, rather than the 4 that we had before. So this is why things are taking forever. I emailed them, let's see.

I think you might have looked at the queue when just one of them was running, but it fluctuates from 0-4 (see OP). It's almost as if the workers are shared with another queue that's not visible (and so, on finishing one PR, the 4 workers sometimes move somewhere else, and after finishing there, come back one by one).

[TomAugspurger]

They sent out an email yesterday explaining some things. They ran into rate limits with their cloud providers.

…

On Thu, Sep 20, 2018 at 2:13 AM h-vetinari ***@***.***> wrote: We somehow got bumped to 1 container, rather than the 4 that we had before. So this is why things are taking forever. I emailed them, let's see. I think you might have looked at the queue when just one of them was running, but it fluctuates from 0-4 (see OP). It's almost as if the workers are shared with another queue that's not visible (and so, on finishing one PR, the 4 workers move somewhere else, and after finishing there, come back one by one). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#22763 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIgGPMXALaPvkhwTwipD5o0rczAcUks5uc0AMgaJpZM4WwLQ4> .

[h-vetinari]

It seems that also the "cancel-on-newer-commit" thing is not working. Right now, I see the jobs for an old commit of #22736 starting (https://circleci.com/gh/pandas-dev/pandas?page=3), even though newer commits exist (https://circleci.com/gh/pandas-dev/pandas?page=1).

[h-vetinari]

Aaaand we're back on the fifth page... https://circleci.com/gh/pandas-dev/pandas?page=4

[TomAugspurger]

What are people's thoughts on dropping circle? The main gaps that would leave in our coverage would be - Just travis for SQL tests (OK, I think) - Fewer tests in other locale's (we have a travis job that uses `zh_CN.UTF-8`. - A linux-based python 3.5 job (OK, to lose, still have 3.5 on our Mac build)

…

On Wed, Oct 3, 2018 at 6:03 PM h-vetinari ***@***.***> wrote: Aaaand we're back on the fifth page... https://circleci.com/gh/pandas-dev/pandas?page=4 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#22763 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIgY0KOVvB0bZ8cvt8nAwjX_T3DWHks5uhUJCgaJpZM4WwLQ4> .

[jreback]

we will inevitable have hard to debug regressions if we lose the locale job - maybe can set this on another job though

the 3.5 is prob somewhat duplicative

[TomAugspurger]

Right. I forgot to finish my comment.

Of those, the hardest to replicate on azure is the SQL tests. The best way to do that would be a docker-based setup where we have Postgres and mysql services that the test container talks to. Azure-CI supports that, but based on 5 minutes of reading there docs, would require a fair amount of effort to setup.

The other (3.5 and locale) can be ported directly to a linux test on azure.

[WillAyd]

Interested in learning more about this so if there’s anything I can do to help lmk

…

Sent from my iPhone

On Oct 4, 2018, at 4:19 AM, Tom Augspurger ***@***.***> wrote: Right. I forgot to finish my comment. Of those, the hardest to replicate on azure is the SQL tests. The best way to do that would be a docker-based setup where we have Postgres and mysql services that the test container talks to. Azure-CI supports that, but based on 5 minutes of reading there docs, would require a fair amount of effort to setup. The other (3.5 and locale) can be ported directly to a linux test on azure. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[h-vetinari]

@TomAugspurger

Why not leave one build in CircleCI? With 4 workers, that will almost be 4x as fast as currently, which should be enough to keep pace with travis/azure. Maybe that job could even combine 3.5/linux/local/SQL tests.

[h-vetinari]

Speed has increased again recently. Down from ~30min average to ~20min.

[h-vetinari]

Runtime has stayed much better (still ~20min), but builds are not cancelled (even if still in the queue) if newer commits are pushed. Having lots of that in #23062 ATM