CircleCI build time varies from 7 to 40 minutes for the same commit

[nponeccop]

Somehow there is big difference in build duration between different ghc versions.
So we need to double-check it's not a cache system malfunction or something.

[alanz]

I think the machines the tests run on are multi-tenant, so it also depends on the load of whatever else is running, that we cannot know.

[nponeccop]

In the build I'm looking at the difference is not caused by noisy neighbors:

• https://circleci.com/gh/haskell/haskell-ide-engine/7213 (slow one, 45 minutes)

  • 4 minutes rebuilding dependencies,
  • 10 minutes creating cache,
  • 10 minutes creating another cache seemingly with the same contents but different key

• https://circleci.com/gh/haskell/haskell-ide-engine/7202 (fast one, 7 minutes) is essentially a no-op:

  • 1 second rebuilding dependencies (because there are none)
  • 1 second skipping cache creation
  • 1 second skipping cache creation

So far apparently the differences are caused by different number of commits in each build (the fast has 2 and the slow 17), which is normal.

Another issue is long cache rebuilds (20 minutes out of 45 is cache compression). So we may try to make caches more granular.

[lukel97]

For https://circleci.com/gh/haskell/haskell-ide-engine/7213 it looks like the most specific cache was invalidated, so it fell back to one of the more generic caches.
The most specific cache IIRC is invalidated whenever any .cabal or the stack.yaml file changes. Ironically, the caching itself is the slowest part it seems, but hopefully after that the 8.4.3 job shouldn't need to create another cache and it should be more within the 20 minute range.