Parse Server version 2.5.1 and newer leaking memory and eventually dying

[mman]

Issue Description

Running 2.5.0 in production since July 1st and having flat CPU and memory consumption around 200MB. Tried to upgrade to 2.6.2, 2.6.1, 2.6.0, 2.5.3, 2.5.2, 2.5.1, and all are leaking memory rather fast and with a very similar pattern. 1G consumed in about 10 minutes. Looking at the changelog It might be related to the parse-server-push-adapter being bumped to version 2.0 in 2.5.1.

Steps to reproduce

Not sure yet. Need help figuring out what is happening.

Expected Results

Stable memory consumption.

Actual Outcome

Parse Server leaking memory until it dies (gets killed by docker).

Environment Setup

• Server
  • parse-server version 2.5.1, 2.5.2, 2.5.3, 2.6.0
  • Operating System: Ubuntu 16.04
  • Hardware: N/A
  • Localhost or remote server? Digital Ocean

Logs/Trace

No errors or strange messages reported, everything working as usual, just memory increasing.

[mman]

Just to add, reverted back to 2.5.0 and memory consumption is back to stable ~180MB.

[flovilmart]

What version of nodejs are you running on?

[mman]

I'm using docker node image based on Ubuntu 16.04 to rebuild all parse servers fresh:

# node --version
v7.9.0

[flovilmart]

We’ll need heapdumps to measure growth and see if it’s related to parse itself or something else

[mman]

Here are the dependencies from my package.json, I'm using fairly simple setup with only s3 adapter, mailgun for mail, and default push adapter.

Looking at the recent changes I'm aware of push getting upreved and s3 being the one that is under heavy changes as well...

: "index.js",
  "dependencies": {
    "express": "~4.2.x",
    "mailgun-js": "*",
    "stathat": "*",
    "parse-server": "2.5.0",
    "parse-server-s3-adapter": "*",
    "parse-server-mailgun": "^2.0.0"
  },

[mman]

I have checked this again, especiall in relation to #4238 and figured out that my docker env got somewhat out of sync and has been running 8.6.0 somewhere along with 7.9.0 elsewhere. Not sure where 7.9.0 came from as node official page does not list it anymore...

I have now streamlined all my custom docker images to be based on node v6.11.4 which I understand is what most folks should use (https://hub.docker.com/_/node/), I'll keep it running over the weekend to observe memory consumption on 2.5.0 and on Monday I'll upgrade to 2.6.3 to see what it feels like on node 6.11.4.

What node version are you using @flovilmart ?

[flovilmart]

We're currently running 6.11.1.

[flovilmart]

And this is our memory usage, as you can see, we have spikes etc.. but usual consumption is around 150mb

[mman]

Just to confirm node@v6.11.4 and parse-server 2.5.0 super stable over the weekend, just tried to rebuild everything with parse-server 2.6.3 and both my experimental instances died after about 20 minutes on memory exhausted:

The only suspicious error message I was able to find was:

Oct  9 17:27:13 XXXX 2b641e9a52e3[1316]: (node:15) Warning: Possible EventEmitter memory leak detected. 11 wakeup listeners added. Use emitter.setMaxListeners() to increase limit

[flovilmart]

Are you able to capture heapdumps in order to identify where the memory is getting swallowed?

[mman]

I will be happy to, please copy paste here a command for me to execute and I'll send it immediately... (read: I'm node newbie)...

[flovilmart]

@mman
there's a few steps involved:

• Add heapdump as a dependency (npm install --save heapdump)
• Add heapdump to your cloud code main (require('heapdump'))
• trigger a snapshot with kill -USR2 PID where PID is the PID of the node process
• Take a few of those and use chrome inspector to see the memory growth.

more infos here: https://www.npmjs.com/package/heapdump

I don't recommend you send them over as it will contain the contents of the memory (dbURL, masterKey etc...) so it's unsafe to post them publicly :)

[mman]

@flovilmart Got the snapshots few minutes apart, one at 119MB the other at 279MB. I'm trying to make sense of them and if I look at objects allocated between the two snapshots, the chrome dev tools are showing me a lot of Object and (array) instances that are referenced from InMemoryCache.CacheController.InMemoryCacheAdapter.LRUCache.Map.Node.Entry.

Looking at 2.5.1 and your change #3979 I see a weak connection here but don't have enough knowledge to speculate.

Would you be willing to take a look at the dumps if I share them with you personally?

thanks,
Martin

[flovilmart]

SO this is the in memory cache, which makes sense that it grows, however, it should naturally stabilize depending on the parameters (max number of keys, ttl etc...) what is the RAM allowance on your instances?

[mman]

Here is the detailed screenshot: