SDK doesn't appear to use Retry-After on 429s

[gavinbarron]

Bug Report

Prerequisites

• Can you reproduce the problem?
• Are you running the latest version?
• Are you reporting to the correct repository?
• Did you perform a cursory search?

Description

During testing of a tool to simulate 429s on the network layer (i.e. outside of the JavaScript library) it was observed that a when a Retry-After or 5s was returned to the client application the SDK issued a retry request before the 5s window had elapsed

Console Errors: N/A

Screenshots:

Steps to Reproduce

1. Use the ChaosProxy sample using the sdk and with the proxy running with 99% failure rate and only 429s as the allowed errors
2. see that the a retry is sent before the 5s wait period is finished

Expected behavior: [What you expected to happen]
SDK should not be retrying until after the number of seconds provided in the Retry-After header

Actual behavior: [What actually happened]
SDK retries the request at 3s and eventually succeeds due to the exponential backoff behavior

Additional Context

Add any other context about the problem here..
Chaos Proxy is here: https://github.com/microsoftgraph/msgraph-chaos-proxy (Microsoft private repo at present)

Usage Information

Request ID - N/A

SDK Version - script link to https://cdn.jsdelivr.net/npm/@microsoft/microsoft-graph-client/lib/graph-js-sdk.js so I assume latest

• Browser (Check, if using Browser version of SDK)

Browser Name - Edge

Version - 105

[waldekmastykarz]

This seems to be caused by the fact that the retry-after header is not listed among the safe headers in the access-control-expose-headers response header coming from Graph.

Because calling Graph from client-side apps is a CORS request, clients are only allowed to access a predefined set of safe headers, plus additional headers defined by the server as safe using the access-control-expose-headers header. Right now, retry-after is not listed among these headers, which is why the fetch API always returns its value as null.

To fix this issue, we'd need to add the retry-after header to the access-control-expose-headers header on Graph.

[sebastienlevert]

Quick update on this issue. We are working with the AGS team to add this header to the access-control-expose-headers. When available worldwide, we'll update this thread.

[altano]

@sebastienlevert I'm not seeing this header in the response to graph API requests. I should expect to see it, right? (I assume you closing the topic means it was exposed at some point)

I do see retry-after listed in access-control-expose-headers but the header still isn't in the response, as you can see:

[sebastienlevert]

Some APIs might not send a "Retry-After" and the SDK will automatically backoff exponentially.

[altano]

@sebastienlevert are you sure about the exponential backoff? Here's what I'm seeing:

Note the timestamps. Retries seem to be happening every few seconds without any backoff.

[sebastienlevert]

Based on this, it should https://github.com/microsoftgraph/msgraph-sdk-javascript/blob/dev/src/middleware/RetryHandler.ts#L136. What is the version of the SDK you are using?

[altano]

I'm on @microsoft/microsoft-graph-client v3.0.7 (latest).

I realized what is going on: retryAttempts is capping out at 3 because of the default retry limit, and then the request fails. Then my code moves on to the next page and it does the same thing, with retryAttempts going from 0 - 3.

I am trying to fix this in my code by having a much higher maxRetries but I can't figure out how. Do I really have to create a middleware chain that replaces the default RetryHandler with a custom one, and then use that in Client.initWithMiddleware, or is there an easier way to control this?

FWIW I think it would make sense to have a MUCH higher default maxRetries value than 3 for 429 responses that don't include the retry-after header, as the first 3 retries are almost certainly going to also fail.