Default number of jobs is unsafe (cabal-install)

[hasufell]

Just ran cabal new-install pandoc-include-code without really thinking and it blew up my 16gb ram machine very quickly. Earlyoom was even unable to kill any of the processes, so I had to wait 15 minutes for the machine to become responsive again, potentially getting file corruption and break my recent work.

Number of processors is almost always the wrong choice for jobs with GHC. This is something that comes from the gcc world and even there it's sometimes wrong if the objects are too big.

The default should be safe, not fast. We shouldn't make random assumptions about the memory of the users machine.

[alanz]

The file ~/.cabal/config has an entry for the number of CPUs to use, by default all of them

-- jobs: $ncpus

I change this to a fixed number, normally 80% or so of the available CPUs.

jobs: 6

[hasufell]

The file ~/.cabal/config has an entry for the number of CPUs to use, by default all of them

I'm aware this can be configured. This bug is about the default being unsafe and crashing your PC.

[23Skidoo]

0.8 * $ncpus might indeed be a better default.

[hasufell]

0.8 * $ncpus might indeed be a better default.

That is still a guess. The default should be plain and simple: 1.

[phadej]

I don't see how setting jobs: 1 would prevent a single GHC process from eating all the RAM anyway: less luckily, but still possible.

[hvr]

I believe the current default was introduced ~5 years ago in cabal 1.18 and this is the first time (as far as I can remember) I'm hearing complaints about this. Forcing jobs: 1 by default feels like throwing out the baby with the bathwater to me; there must be a better heuristic for inferring a safe upper bound of parallelism (e.g. by factoring in the amount of available memory; possibly combined with passing RTS flags to ghc) to reduce the risk of an OOM than to force >90% of cabal users to tweak their configuration cause jobs:1 is certainly not what the vast majority of cabal users wants... ;-)

[hasufell]

I don't see how setting jobs: 1 would prevent a single GHC process from eating all the RAM anyway: less luckily, but still possible.

It is quite unlikely and has not happened a single time over half a year with projects being compiled regularly: pandoc, amazonka, gogol, docusign-base.

However, with the default, chances are 90% my laptop crashes or the OOM killer is triggered if I am lucky.

Of course, this only happens on a new setup or somewhere else where I forgot to set or change this setting.

Forcing jobs: 1 by default feels like throwing out the baby with the bathwater to me; there must be a better heuristic for inferring a safe upper bound of parallelism (e.g. by factoring in the amount of available memory; possibly combined with passing RTS flags to ghc) to reduce the risk of an OOM than to force >90% of cabal users to tweak their configuration cause jobs:1 is certainly not what the vast majority of cabal users wants... ;-)

You're not really "forcing" anything here. Defaults are not meant to be smart, but safe, imo. If the user sees slow compile times, he may tweak the setting. Until anyone can come up with a safe heuristic (which is quite hard, because there are so many variables), I would say the default should be 1.

[phadej]

Out of curiosity, what's your $ncpus (you have 16GB of RAM, don't you?)

…

Sent from my iPhone

On 10 Dec 2018, at 4.18, Julian Ospald ***@***.***> wrote: I don't see how setting jobs: 1 would prevent a single GHC process from eating all the RAM anyway: less luckily, but still possible. It is quite unlikely and has not happened a single time over half a year with projects being compiled regularly: pandoc, amazonka, gogol, docusign-base. However, with the default, chances are 90% my laptop crashes or the OOM killer is triggered if I am lucky. Forcing jobs: 1 by default feels like throwing out the baby with the bathwater to me; there must be a better heuristic for inferring a safe upper bound of parallelism (e.g. by factoring in the amount of available memory; possibly combined with passing RTS flags to ghc) to reduce the risk of an OOM than to force >90% of cabal users to tweak their configuration cause jobs:1 is certainly not what the vast majority of cabal users wants... ;-) You're not really "forcing" anything here. Defaults are not meant to be smart, but safe. If the user sees slow compile times, he may tweak the setting. Until anyone can come up with a safe heuristic (which is quite unlikely, because there are too many variables), I would say the default should be 1. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[hasufell]

Out of curiosity, what's your $ncpus (you have 16GB of RAM, don't you?)

8 cores
16GB of ram

[amigalemming]

I am annoyed for long by the ubiquitous assumption that $NCPUS is a good default choice for the number of threads, not only in Cabal. Since the advent of multitasking systems ressources must be shared between different processes. Since then it was not ok to eat up all memory by a single application and it was no longer ok to do busy waits, because there are other programs running on the machine and now for the same reasons it is not ok to occupy all cores without permission.
On my machine the number of cores is virtually 8 because of hyperthreading. But the hyperthreads practically don't do any speedup, they only eat up memory and make harddisk go crazy. Even 4 cores are not the optimum because of heavy harddisk load. i usually choose -j2 and --ghc-option=-j2.
A Raspberry Pi has 4 cores and 1 GB of RAM. With the default "jobs" configuration you turn quickly into thrashing.
Please stop eating up all my machine ressources without asking, please, please, please!

[gbaz]

I'd be keen on 0.5 * ncpus as a good default, even on a desktop machine where a user is perhaps multitasking while compiling.

[amigalemming]

I don't see how setting jobs: 1 would prevent a single GHC process from eating all the RAM anyway: less luckily, but still possible.

We can't choose smaller than 1, can we?

[amigalemming]

I'd be keen on 0.5 * ncpus as a good default, even on a desktop machine where a user is perhaps multitasking while compiling.

This does not address the initially made point: If the number of jobs is too big, you lose control over your machine. If the number of jobs is too small, you can increase the number until it fits your needs.