Idea to minimize memory fingerprint

[oesteban]

Summary

Large workflows are prone to get killed by the OOM Killer in linux and similar devices in other systems. The common denominator is that the RSS memory fingerprint is sensible, not too big, but the VM skyrockets when the workflow has many nodes.

Actual behavior

Except for pure python interfaces, all nipype interfaces fork new processes via subprocess.Popen. That way, all new processes are started using fork and allocate for twice as much virtual memory as it is allocated before forking. This, in addition with python inefficiency to garbage collect after finishing the process, leads to overcommitting memory (and processes being killed on systems that would not allow that).

Expected behavior

Less memory consumption

How to replicate the behavior

Run fmriprep on a large dataset, disallowing overcommitting and with a low memory limit (e.g. 8GB)

Idea:

Patch subprocess.Popen with multiprocessing.context.Popen. Doing that, in theory, all these processes should be forked on the server process (which should have a constant, small memory fingerprint).

WDYT @satra @effigies @chrisfilo ?

BTW, this solution might be only possible with Python>=3.4. As per https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods:

Changed in version 3.4: spawn added on all unix platforms, and forkserver added for some unix platforms. Child processes no longer inherit all of the parents inheritable handles on Windows.

[chrisgorgo]

Sounds reasonable. By "Patch" you mean replace?

…

On Tue, Nov 13, 2018 at 10:42 AM Oscar Esteban ***@***.***> wrote: Summary Large workflows are prone to get killed by the OOM Killer in linux and similar devices in other systems. The common denominator is that the RSS memory fingerprint is sensible, not too big, but the VM skyrockets when the workflow has many nodes. Actual behavior Except for pure python interfaces, all nipype interfaces fork new processes via subprocess.Popen. That way, all new processes are started using fork and allocate for twice as much virtual memory as it is allocated before forking. This, in addition with python inefficiency to garbage collect after finishing the process, leads to overcommitting memory (and processes being killed on systems that would not allow that). Expected behavior Less memory consumption How to replicate the behavior Run fmriprep on a large dataset, disallowing overcommitting and with a low memory limit (e.g. 8GB) Idea: Patch subprocess.Popen with multiprocessing.context.Popen. Doing that, in theory, all these processes should be forked on the server process (which should have a constant, small memory fingerprint). WDYT @satra <https://github.com/satra> @effigies <https://github.com/effigies> @chrisfilo <https://github.com/chrisfilo> ? BTW, this solution might be only possible with Python>=3.4. As per https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods : *Changed in version 3.4*: spawn added on all unix platforms, and forkserver added for some unix platforms. Child processes no longer inherit all of the parents inheritable handles on Windows. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2776>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAOkp9S8tpPqYViC3P9VhnlIZXQw_j4lks5uuxJ6gaJpZM4YcRYq> .

[effigies]

Seems worth trying.

[satra]

let's try it out.

are we the only ones struggling with memory? how does dask handle this?

[oesteban]

@chrisfilo: By "Patch" you mean replace?
@effigies: Seems worth trying.
@satra let's try it out.

Yes, although I'm checking and such replacement is not as easy as I first thought. The interface of the Popen in multiprocessing is not the same as that in subprocess.

@satra are we the only ones struggling with memory? how does dask handle this?
I think this is heavily related to this: https://bugs.python.org/issue8713#msg105719, which gave rise to adding the forkserver mode in multiprocessing.

I think there is one big difference w.r.t. dask: they don't ubiquitously use subprocess.Popen (we do with every CommandLineInterface) which uses this fork+exec mode of linux. Since they just run pickleable functions that are sent to the workers, the sub-processes already exist and they are only re-initiated if maxtasksperchild is reached. If we were able to run fmriprep with dask I'd say we would see the same issues.

My impression is that this issue only happens with linux.

[oesteban]

related: https://bugs.python.org/issue20104#msg222570

EDIT: this post in the same thread may also apply https://bugs.python.org/issue20104#msg289758

[oesteban]

This would be an alternative (python 3.8, not sure how easy to backport) https://docs.python.org/3.8/library/os.html#os.posix_spawn

[effigies]

There is also the 'spawn' multiprocessing context. Maybe the context could become a plugin argument, and thus heavy pipelines like fMRIPrep could use that without significantly changing the performance of existing pipelines.

[oesteban]

Yep, the problem is that subprocess.Popen is totally independent of multiprocessing.

I would not expect a great difference between forkserver and spawn if we managed to create such patch. Although, the more I read, the clearer I see we really need spawn here instead of fork.

[oesteban]

This might be too big to try. Just in case I wrote this bpo https://bugs.python.org/issue35238?@ok_message=msg%20329868%20created%0Aissue%2035238%20created&@template=item#msg329868
Let's see the reactions.

[oesteban]

I have a functional implementation of nipype running commands through posix_spawn.

Patching it into a custom fmriprep docker image**, I've gotten the following virtual memory profile:

Using the same docker image without patching nipype (i.e., using fork_exec):

** Docker image was built with latest fmriprep's master, with the addition of memory_profiler. The tool was modified to measure VMS instead of RMS.

Conclusion: I'm either measuring wrongly*** or changing to spawn does not have any effect. Another possibility is that ds000005/sub-01 is not a big enough dataset so as to trigger memory problems.

*** It could be because my patch to memory_profiler is faulty or because memory_profiler is just not the right tool to measure this. Or that sampling at 0.01s rate is not fast enough to capture the memory duplication of fork and thus allocated memory gets freed before you can even measure.

Extra thought: I'll dig deeper on a peak of 200GB that happens soon after starting fmriprep, as seen on the same profiles without zooming in:

Spawn:

Fork:

[oesteban]

Apparently, switching to Python 3.7 (incl. #2778, thus with forkserver mode process pool) seems to reduce memory usage by 1~2GiB in this test case:

UPDATE: corresponding plot using posix_spawn:

[effigies]

So spawn appears to do very little. It looks like it's the fork of the main process that's killing us, not the secondary Popen fork. Correct?

And spawn looks noticeably slower. Note that the transitions that happen around 2000s and 4000s with Popen occur around 2200s and 4800s with spawn.

[oesteban]

Correct

EDIT: Correct iff we are are not missing out memory allocations of fork_exec because they happen very fast (way below the sampling frequency of 0.01s) or because they don't add to the VMS.

[effigies]

Ah, good point. Maybe we can test by allocating ~1/4 of available memory in the base process and setting the overcommit policy to disallow?