Fix flaky tests

[mtrezza]

New Issue Checklist

• I am not disclosing a vulnerability.
• I am not just asking a question.
• I have searched through existing issues.
• I can reproduce the issue with the latest version of Parse Server.

Issue Description

Flaky tests impede the CI process. With the increasing number of test scenarios to ensure Parse Server compatibility with recent versions of MongoDB, Postgres and Node.js, the likeliness of flaky tests disrupting the CI process also increases. In addition, GitHub Actions currently does not support to re-run individual jobs which makes the manual re-run a tedious effort.

Expected outcome

Identify root cause of flakiness and fix for tests to be stable.

The following lists flaky tests to analyze and possibly find a pattern to determine the root cause. The list should also be a helper for maintainers to identify flaky tests when reviewing PRs. Feel free to update the list if you encounter a flaky test that is not in the list yet.

---

Last list update: 2021-02-24

Flaky tests

Tests which still have to be analyzed to fix the flakiness.

Schema Performance test schema update class

Error: Could not add field fooOne

Installations update ios fails with new deviceToken and no installationId

  Message:
    Failed
  Stack:
    Error: Failed
        at <Jasmine>
        at UserContext.<anonymous> (/home/runner/work/parse-server/parse-server/spec/AuthenticationAdapters.spec.js:1671:7)
        at <Jasmine>
        at processTicksAndRejections (internal/process/task_queues.js:97:5)

Parse.User testing only creates a single session for an installation / user pair (#2885)

 Message:
  Expected 2 to be 1.
Stack:
  Error: Expected 2 to be 1.
      at <Jasmine>
      at query.find.then.results (/home/runner/work/parse-server/parse-server/spec/ParseUser.spec.js:1294:32)
      at process._tickCallback (internal/process/next_tick.js:68:7)
Message:
  Expected 'r:a49facd19f31028da42c3058ce5c35a3' to be 'r:0bcc75ade014c3ecbc500a5a29606929'.
Stack:
  Error: Expected 'r:a49facd19f31028da42c3058ce5c35a3' to be 'r:0bcc75ade014c3ecbc500a5a29606929'.
      at <Jasmine>
      at query.find.then.results (/home/runner/work/parse-server/parse-server/spec/ParseUser.spec.js:1295:48)
      at process._tickCallback (internal/process/next_tick.js:68:7)

• Observed in environment(s): Postgres 12, Postgis 3.0

ParseLiveQuery expect afterEvent delete

 Message:
  Failed
Stack:
  Error: Failed
      at <Jasmine>
      at UserContext.it (/home/runner/work/parse-server/parse-server/spec/AuthenticationAdapters.spec.js:1671:7)
      at process._tickCallback (internal/process/next_tick.js:68:7)

• Observed in environment(s): Node 10

Apple Game Center Auth adapter validateAuthData should validate

Message:
  Error: Timeout - Async function did not complete within 20000ms (set by jasmine.DEFAULT_TIMEOUT_INTERVAL)
Stack:
  Error: Timeout - Async function did not complete within 20000ms (set by jasmine.DEFAULT_TIMEOUT_INTERVAL)
      at <Jasmine>
      at ontimeout (timers.js:436:11)
      at tryOnTimeout (timers.js:300:5)
      at listOnTimeout (timers.js:263:5)
      at Timer.processTimers (timers.js:223:10)

• Observed in environment(s): Node 10

Cloud Code test beforeSave with invalid field

  Message:
    Failed: deadlock detected
  Stack:
    error properties: Object({ length: 317, severity: 'ERROR', code: '40P01', detail: 'Process 151 waits for AccessExclusiveLock on relation 50837 of database 19568; blocked by process 156.
    Process 156 waits for AccessShareLock on relation 50712 of database 19568; blocked by process 151.', hint: 'See server log for query details.', position: undefined, internalPosition: undefined, internalQuery: undefined, where: undefined, schema: undefined, table: undefined, dataType: undefined, constraint: undefined, file: 'deadlock.c', routine: 'DeadLockReport' })
        at <Jasmine>
        at Parser.parseErrorMessage (/home/runner/work/parse-server/parse-server/node_modules/pg-protocol/dist/parser.js:278:15)
        at Parser.handlePacket (/home/runner/work/parse-server/parse-server/node_modules/pg-protocol/dist/parser.js:126:29)
        at Parser.parse (/home/runner/work/parse-server/parse-server/node_modules/pg-protocol/dist/parser.js:39:38)
        at Socket.stream.on (/home/runner/work/parse-server/parse-server/node_modules/pg-protocol/dist/index.js:10:42)
        at Socket.emit (events.js:198:13)
        at Socket.EventEmitter.emit (domain.js:448:20)
        at addChunk (_stream_readable.js:288:12)
        at readableAddChunk (_stream_readable.js:269:11)
        at Socket.Readable.push (_stream_readable.js:224:10)
        at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)

rest create test specified session length

Message:
  Expected 28 to equal 27.
Stack:
  Error: Expected 28 to equal 27.
  at rest.create.then.then.r (/home/runner/work/parse-server/parse-server/spec/rest.spec.js:602:37)
  at process._tickCallback (internal/process/next_tick.js:68:7)

• Observed in environment(s): ?

ParseGraphQLServer Auto API Schema Objects Queries Get read preferences should support includeReadPreference argument

Error: GraphQL error: Object not found.

ParseGraphQLServer Auto API Schema Objects Queries Find read preferences should support readPreference argument

Message:
  Expected false to be true.
Stack:
  Error: Expected false to be true.
  at UserContext. (/home/runner/work/parse-server/parse-server/spec/ParseGraphQLServer.spec.js:5512:52)
  at processTicksAndRejections (internal/process/task_queues.js:93:5)

• Observed in environment(s): ?

ParseGraphQLServer Auto API Schema Files Mutations Create should return File object

Message:
  TypeError: Cannot read property 'clientMutationId' of null
Stack:
  at UserContext. (/home/runner/work/parse-server/parse-server/spec/ParseGraphQLServer.spec.js:6603:43)
  at processTicksAndRejections (internal/process/task_queues.js:97:5)

Possibly fixed

Tests for which a PR has been merged to fix the flakiness. These are currently under observation to confirm the fix. If you observe such a test to fail, please post a comment for further evaluation.

by #7187:

ParseServerRESTController transactions should handle a batch request with transaction = true

Unhandled promise rejection: [object Object],[object Object]

• Observed in environment(s):
  • Mongo 4.2, ReplicaSet, WiredTiger, Node 10
  • Mongo 4.0, ReplicaSet, WiredTiger, Node 14

by #7196:

Cloud Code cloud jobs should define a job

Unhandled promise rejection: ParseError: 101 Object not found.

• Observed in environment(s):
  • Mongo 4.4, ReplicaSet, WiredTiger, Node 14

Cloud Code cloud jobs should define a job

Message:
  Unhandled promise rejection: error: relation "_JobStatus" does not exist (line 1180)
Stack:
  error properties: Object({ length: 108, severity: 'ERROR', code: '42P01', detail: undefined, hint: undefined, position: '8', internalPosition: undefined, internalQuery: undefined, where: undefined, schema: undefined, table: undefined, dataType: undefined, constraint: undefined, file: 'parse_relation.c', routine: 'parserOpenTable' })
  at <Jasmine>
  at Parser.parseErrorMessage (/home/runner/work/parse-server/parse-server/node_modules/pg-protocol/dist/parser.js:278:15)
  at Parser.handlePacket (/home/runner/work/parse-server/parse-server/node_modules/pg-protocol/dist/parser.js:126:29)
  at Parser.parse (/home/runner/work/parse-server/parse-server/node_modules/pg-protocol/dist/parser.js:39:38)
  at Socket.stream.on (/home/runner/work/parse-server/parse-server/node_modules/pg-protocol/dist/index.js:10:42)
  at Socket.emit (events.js:198:13)
  at Socket.EventEmitter.emit (domain.js:448:20)
  at addChunk (_stream_readable.js:288:12)
  at readableAddChunk (_stream_readable.js:269:11)
  at Socket.Readable.push (_stream_readable.js:224:10)
  at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)

• Observed in environment(s):
  • Postgres 11, Postgis 3.1, Node 14.15.5

Cloud Code cloud jobs should run with master key basic auth

Message:
  Unhandled promise rejection: ParseError: 101 Object not found.
Stack:
  error properties: Object({ code: 101 })
  Error: Object not found.
      at /home/runner/work/parse-server/parse-server/lib/Controllers/DatabaseController.js:49:3084
      at <Jasmine>
      at processTicksAndRejections (internal/process/task_queues.js:93:5)

• Observed in environment(s):
  • Redis Cache, Node 14.15.5

by #7212:

RedisCacheAdapter should find un-expired records

Message:
  Expected null to equal 'world'.
Stack:
  Error: Expected null to equal 'world'.
  at /home/runner/work/parse-server/parse-server/spec/RedisCacheAdapter.spec.js:97:36
  at processTicksAndRejections (internal/process/task_queues.js:93:5)
Message:
  Expected null not to equal null.
Stack:
  Error: Expected null not to equal null.
  at /home/runner/work/parse-server/parse-server/spec/RedisCacheAdapter.spec.js💯40
  at processTicksAndRejections (internal/process/task_queues.js:93:5)

bv #7213:

PushController properly creates _PushStatus

Message:
  Expected 'pending' to be 'running'.
Stack:
  Error: Expected 'pending' to be 'running'.
  at reconfigureServer.then.then.then.then.then.catch.then (/home/runner/work/parse-server/parse-server/spec/PushController.spec.js:618:48)
at process._tickCallback (internal/process/next_tick.js:68:7)

by #7228:

PostgresStorageAdapter should use index for caseInsensitive query using default indexname

  Message:
    error: duplicate key value violates unique constraint "_User_pkey" (line 434)
  Stack:
    error properties: Object({ length: 219, severity: 'ERROR', code: '23505', detail: 'Key ("objectId")=(d3c9d0cbebeeb2efcc62a95f09255ff2) already exists.', hint: undefined, position: undefined, internalPosition: undefined, internalQuery: undefined, where: undefined, schema: 'public', table: '_User', dataType: undefined, constraint: '_User_pkey', file: 'nbtinsert.c', routine: '_bt_check_unique' })
        at <Jasmine>
        at Parser.parseErrorMessage (/home/runner/work/parse-server/parse-server/node_modules/pg-protocol/dist/parser.js:278:15)
        at Parser.handlePacket (/home/runner/work/parse-server/parse-server/node_modules/pg-protocol/dist/parser.js:126:29)
        at Parser.parse (/home/runner/work/parse-server/parse-server/node_modules/pg-protocol/dist/parser.js:39:38)
        at Socket.stream.on (/home/runner/work/parse-server/parse-server/node_modules/pg-protocol/dist/index.js:10:42)
        at Socket.emit (events.js:198:13)
        at Socket.EventEmitter.emit (domain.js:448:20)
        at addChunk (_stream_readable.js:288:12)
        at readableAddChunk (_stream_readable.js:269:11)
        at Socket.Readable.push (_stream_readable.js:224:10)
        at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)

by #7265:

Parse.GeoPoint testing creating geo point exception two fields

Message:
  Failed: deadlock detected
Stack:
  error properties: Object({ length: 319, severity: 'ERROR', code: '40P01', detail: 'Process 150 waits for AccessExclusiveLock on relation 139374 of database 19581; blocked by process 154.
  Process 154 waits for AccessShareLock on relation 139249 of database 19581; blocked by process 150.', hint: 'See server log for query details.', position: undefined, internalPosition: undefined, internalQuery: undefined, where: undefined, schema: undefined, table: undefined, dataType: undefined, constraint: undefined, file: 'deadlock.c', routine: 'DeadLockReport' })
      at <Jasmine>
      at Parser.parseErrorMessage (/home/runner/work/parse-server/parse-server/node_modules/pg-protocol/dist/parser.js:278:15)
      at Parser.handlePacket (/home/runner/work/parse-server/parse-server/node_modules/pg-protocol/dist/parser.js:126:29)
      at Parser.parse (/home/runner/work/parse-server/parse-server/node_modules/pg-protocol/dist/parser.js:39:38)
      at Socket.stream.on (/home/runner/work/parse-server/parse-server/node_modules/pg-protocol/dist/index.js:10:42)
      at Socket.emit (events.js:198:13)
      at Socket.EventEmitter.emit (domain.js:448:20)
      at addChunk (_stream_readable.js:288:12)
      at readableAddChunk (_stream_readable.js:269:11)
      at Socket.Readable.push (_stream_readable.js:224:10)
      at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)

• Observed in environment(s): Postgres

by #7267:

Cloud Code cloud jobs should set the message / success on the job

Message:
  Expected 'running' to equal 'succeeded'.
Stack:
  Error: Expected 'running' to equal 'succeeded'.
      at <Jasmine>
      at promise.then.then.jobStatus (/home/runner/work/parse-server/parse-server/spec/CloudCode.spec.js:1666:45)
      at process._tickCallback (internal/process/next_tick.js:68:7)

PushController should not enqueue push when device token is not set

  Message:
    Expected 'running' to be 'succeeded'.
  Stack:
    Error: Expected 'running' to be 'succeeded'.
        at <Jasmine>
        at query.find.then.results (/home/runner/work/parse-server/parse-server/spec/PushController.spec.js:1128:44)
        at process._tickCallback (internal/process/next_tick.js:68:7)

Parse.Push does not get stuck with _PushStatus 'running' on 1 installation added

Message:
  Expected 'running' to equal 'succeeded'.
Stack:
  Error: Expected 'running' to equal 'succeeded'.
      at <Jasmine>
      at reconfigureServer.then.then.then.then.then.results (/home/runner/work/parse-server/parse-server/spec/Parse.Push.spec.js:388:38)
      at process._tickCallback (internal/process/next_tick.js:68:7)
Message:
  Expected 1 to equal undefined.
Stack:
  Error: Expected 1 to equal undefined.
      at <Jasmine>
      at reconfigureServer.then.then.then.then.then.results (/home/runner/work/parse-server/parse-server/spec/Parse.Push.spec.js:390:37)
      at process._tickCallback (internal/process/next_tick.js:68:7)

Fixed tests

Tests that have not been observed anymore for a while and are therefore considered fixed.

[mtrezza]

@dplewis Do you have any idea how we can investigate this? Maybe add NewRelic to it to get the call stack when a test fails?

[davimacedo]

@dplewis Do you have any idea how we can investigate this? Maybe add NewRelic to it to get the call stack when a test fails?

The way I do is basically creating a loop around the test and making it to execute let's say 100 times. By this way you can see it failing and you can debug. I am tackling the second one now.

[davimacedo]

#7196 probably solves items 2 and 3

[davimacedo]

Few more flaky tests that I've observed:

1. rest create test specified session length
   49
   Message:
   50
   Expected 28 to equal 27.
   51
   Stack:
   52
   Error: Expected 28 to equal 27.
   53
   at
   54
   at rest.create.then.then.r (/home/runner/work/parse-server/parse-server/spec/rest.spec.js:602:37)
   55
   at process._tickCallback (internal/process/next_tick.js:68:7)

2. Parse.GeoPoint testing creating geo point exception two fields
   49
   Message:
   50
   Failed: deadlock detected
   51
   Stack:
   52
   error properties: Object({ length: 319, severity: 'ERROR', code: '40P01', detail: 'Process 158 waits for AccessExclusiveLock on relation 139308 of database 19596; blocked by process 152.
   53
   Process 152 waits for AccessShareLock on relation 139183 of database 19596; blocked by process 158.', hint: 'See server log for query details.', position: undefined, internalPosition: undefined, internalQuery: undefined, where: undefined, schema: undefined, table: undefined, dataType: undefined, constraint: undefined, file: 'deadlock.c', routine: 'DeadLockReport' })
   54
   at
   55
   at Parser.parseErrorMessage (/home/runner/work/parse-server/parse-server/node_modules/pg-protocol/dist/parser.js:278:15)
   56
   at Parser.handlePacket (/home/runner/work/parse-server/parse-server/node_modules/pg-protocol/dist/parser.js:126:29)
   57
   at Parser.parse (/home/runner/work/parse-server/parse-server/node_modules/pg-protocol/dist/parser.js:39:38)
   58
   at Socket.stream.on (/home/runner/work/parse-server/parse-server/node_modules/pg-protocol/dist/index.js:10:42)
   59
   at Socket.emit (events.js:198:13)
   60
   at Socket.EventEmitter.emit (domain.js:448:20)
   61
   at addChunk (_stream_readable.js:288:12)
   62
   at readableAddChunk (_stream_readable.js:269:11)
   63
   at Socket.Readable.push (_stream_readable.js:224:10)
   64
   at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)

3. ParseGraphQLServer Auto API Schema Objects Queries Find read preferences should support readPreference argument
   22
   Message:
   23
   Expected false to be true.
   24
   Stack:
   25
   Error: Expected false to be true.
   26
   at
   27
   at UserContext. (/home/runner/work/parse-server/parse-server/spec/ParseGraphQLServer.spec.js:5512:52)
   28
   at
   29
   at processTicksAndRejections (internal/process/task_queues.js:93:5)

[mtrezza]

I added them to the list above. I'll mark tests as solved by #7196 if they don't occur anymore after some time.

[davimacedo]

"ParseServerRESTController transactions should handle a batch request with transaction = true" was actually also solved by #7187 so let's observe this one as well.

[mtrezza]

Thanks, updated.