Processing time keep increasing everyday

[tobernguyen]

I used NewRelic for monitoring parse-server performance. As I observed, the amount of time taken in NodeJS processing keep increasing over time. Average NodeJS processing time is 30ms for the first day (after restarting parse-server). Then after the second day, the NodeJS processing is now about 50ms. If I leave it running through the 3rd day, it will increase to 70 ~ 100ms.

But if I restart the server, processing time will decrease to 30ms and keep increasing over time. My instance's info:

• CPU: 2 core
• Ram: 2GB

Parse-server has never used all 2gb ram, and the memory it took very stable over time as well as the CPU usage (about 50% ram and 50% CPU).

So the question is: how can I know what cause the increasing of processing time over time? And how to resolve that problem, which made me have to restart parse-server every day. Anyone experience same problem?

[flovilmart]

Do you have a trace of the request times, min, max etc..

[tobernguyen]

@flovilmart You mean get it from NewRelic?

[huang1196]

@flovilmart

I have the same problem. Our parse-server running on an ec2-instance with ELB. According to the elb monitoring, the response time is increasing everyday. The first day's average lantency is around 40ms while the 6th day's average latency is around 1000ms. After a restart, the latency would be back to 40ms.

According to the CPU utilization of my aws ec2-instance( C4 Xlarge), it is 1% ~ 2% when I restart the parse-server ( 40~70 req/min). But after 6 days, the average cpu utilization per minute will be 20% ~ 40% even 50%. The RAM usage is stable, around 200MB.

[benitech]

i would assume it to be more of a MongoDB response time issue than the parse server. Parse is pretty much a standard nodejs/express app and should behave accordingly.

[tobernguyen]

@benitech why restart Parse Server resolved the problem? (and of course not restart MongoDB server)

[benitech]

i am only looking at this from my past experience running node.js apps.

Can you try running multiple threads using PM2? and see if you experience the same issue on any thread? You should be able to run 2 threads based on your CPU, but might need to increase total RAM to 3GB.

[airdrummingfool]

I am also experiencing this issue. I run 2x m3.medium instances on AWS behind a load balancer that serve a consistent 20-30 req/sec. Over the course of about 20 hours the node process' CPU usage slowly grows from its base around 25%. The first time it took me a while to catch, and CPU usage grew to 100%; the second time I had a CloudWatch alarm and I restarted the process at 60% usage. Request latency (I believe this is between the ELB and the EC2 instances) grows similarly, though not drastically until the CPU hits 100%.

Below is a picture of an instance CPU usage vs request count from the ELB. I've highlighted the two steep drops in CPU usage that occurred when I manually restarted the node process.

[nil8tor]

Agreed. Same problem here. We are running 10 app servers, all same code, and all of them have to have reboots scheduled for every 72 hours or performance goes to crap. It has been this way from 2.1.6 to 2.2.7. Not that is matters, but we are load balancing between all servers and each server runs node in a localized cluster for a combined 80 CPU cores. Even in this scenario and combined utilization rarely exceeding 25%, we still have to perform reboots for performance to come back.

[araskin]

This is interesting thread. Forgive me if I am off base (I am fairly new to Node) but I would think this is more likely to be a Node issue rather then Parse. No?

[blacha]

This is going to be really tough to track down without having a simple test case that causes this.

I have a couple of applications running across multiple hosts which are not showing this behaviour.

The first places I would look would be into resource usage, how is the disk and memory usage on those machines? If they are running low on either it could cause the CPU usage to grow.

Then also in cloud code, In Parse.com, functions that were left around after the completion of a request were destroyed, in parse-server these functions will be left around, so if you have any setTimeout, or setInterval code being run, it could build up that way.

If you could also look to see if you can find a simple test case where it is shows this behaviour that would be really helpful.

[airdrummingfool]

I added New Relic monitoring to my servers, so I should be able to get hard numbers the next time this issue crops up. That being said, I never noticed memory usage approach even 25% on my instances, and my disk space usage is next to nothing (<20%).

I don't have any setTimeout or setInterval calls in my code, but I have noticed some 499 responses going out (nginx's way of reporting a client-side timeout), which could indicate functions sticking around after a request is terminated by the client. Is there a way in node to log whenever this happens, or some kind of instrumentation that would measure it?

I'm going to start some local testing to see if I can find a simple test case. I tried hammering a dev instance with ~4 million requests yesterday, but CPU usage stayed very consistent - I guess the particular function I was testing isn't a culprit.

[blacha]

The 499 time out sound very interesting, sounds like one of your cloud functions may be getting stuck

Do you have a way to track which functions are being called when that happens?

I have a really simple logger over all my express routes, which logs the time, method, user and path for all parse requests.

app.use(function(req, res, next) {
    req.$startTime = Date.now();
    var currentUrl = req.url;

    var isParse = currentUrl.indexOf(PARSE_API_URL) === 0;
    if (!isParse) {
        return next();
    }
    res.on('finish', function() {
        var logObj = {
            method: req.method,
            duration: Date.now() - req.$startTime,
            status: res.statusCode,
            $user: req.auth && req.auth.user && req.auth.user.get('username')
        };

        if (res.statusCode >= 500) {
            Log.error(logObj, currentUrl);
        } else if (res.statusCode >= 400) {
            Log.warn(logObj, currentUrl);
        } else {
            Log.trace(logObj, currentUrl);
        }
    });
    next();
});

[airdrummingfool]

Awesome, I've been looking for a logging function to do exactly what yours does - thanks! I've added it in, we'll see if anything interesting comes out of it.

Unfortunately, it won't log 499 errors, since those are thrown by nginx (and show that Parse/express is stuck somewhere). I can at least see those (without details) in my access.log file.

So far, whenever I've run into 499s, it is often because I'm running a query or a save on an object that has an ACL without using useMasterKey: true, and I'm not handling the rejected Promise that is returned. That is certainly my fault, but it seems to me that Parse (or express?) should be able to catch rejected promises that get ignored and error out gracefully instead of getting stuck. There's also at least one instance of a function that 499s even though I useMasterKey: true all of the queries & saves, and also handle all possible rejected promises... that one needs more investigating.

[drew-gross]

Yeah, I thought it might be something like that. We can probably come up with some better behaviour here.