Investigate jobs getting stuck on pending state

[thoven87]

In some instance jobs can get stuck in pending state while they are not present in the job queue table. Investigate what's causing these jobs to end up in this state. So far, I have seen this happening with schedule jobs where some of the "partitions" are skipped. e.g:

enum StationKind: Codable {
    case bbc
    case bloomberg
}

protocol NewsIngestionJobParameters: Codable, Sendable {
    var kind: StationKind { get set }
}

struct BBCNewsIngestionJobParameters: JobParameters, NewsIngestionJobParameters {
    static let jobName: String = "BBCNewsIngestionJob"
    var kind: StationKind = .bbc
}

struct BloombergNewsIngestionJobParameters: JobParameters, NewsIngestionJobParameters {
    static let jobName: String = "BloombergNewsIngestionJob"
    var kind: StationKind = .bloomberg
}

var jobScheduler = JobSchedule()

jobScheduler.addJob(BirthdayRemindersJobParameters(), schedule: .daily(hour: 9, timeZone: .init(abbreviation: "EST")!))
jobScheduler.addJob(ActivityRemindersJobParameters(), schedule: .daily(hour: 9, timeZone: .init(abbreviation: "EST")!))
jobScheduler.addJob(BBCNewsIngestionJobParameters(), schedule: .onMinutes([0, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55]))
jobScheduler.addJob(BloombergNewsIngestionJobParameters(), schedule: .onMinutes([0, 10, 20, 30, 40, 50]))

The Bloomberg job gets skipped on minute 10 for a while and after running the job service between a day to a week, we will get job of type BloombergNewsIngestionJobParameters in the jobs table with state pending while there's no reference of these jobs in the queue table.

This behavior is noticed with a docker container with 512MB 1 CPU where concurrency was set between 3 - 100.

These "Jobs" do nothing other than printing "News From Station"

[adam-fowler]

Have you checked logs to see what has happened to the jobs that get stuck in pending?

[thoven87]

Have you checked logs to see what has happened to the jobs that get stuck in pending?

I will have time this weekend to run more tests.

[thoven87]

I have updated the POC job repo will the "jobs" I have been testing on GCP.

After the running the jobs for a while from 20 workers to 300 plus and running every job with a schedule of everyMinutes, I see the followings the logs

connection aborted - error reading from instance: read tcp 169.254.8.1:41938->35.245.220.30:3307: read: connection reset by peer

There's no MYSQL server running in the that GCP project, it's a Postgres instance that's attached to a cloud run service (this is clearly a google issue) I will also look into Postgres NIO for what happens when a Unix socket connection is lost. Because the container just hangs and never reconnects. In the end we have jobs stuck in either processing state, pending state e.t.c and will not get run unless the container restarts and a user has set to rerun these jobs.

I will switch the cloud run instance to be always on and see if this issue goes away.

[adam-fowler]

The key thing is to look for logs related to jobs that aren't moving from pending. Do you have that information? Are we seeing Postgres connection fails around these?

God knows why there on connections failures on the MySQL port

[adam-fowler]

I had left something running locally and it crashed in the Postgres nio connection pool. I'm going run it in the debugger and catch the crash.

[thoven87]

The key thing is to look for logs related to jobs that aren't moving from pending. Do you have that information? Are we seeing Postgres connection fails around these?

God knows why there on connections failures on the MySQL port

I don't have any information on that since the container was started with log level set to info. I will have a go again later today and will report.