Formulate a long-term HTTP client plan

[jmakeig]

We use Jersey currently for HTTP communication with the MarkLogic REST API. However, the long-term viability of Jersey is uncertain. (TODO: enumerate existing issues.) Let’s put together a plan for a robust HTTP client that will support our current and future needs.

[sammefford]

This is related / overlapping #65 "Upgrade or remove Jersey"

[sammefford]

With the upcoming HTTP/2, some core paradigms may change. Any plans we make should account for that fact.

One example of the impact of HTTP/2 push: DMSDK uses pagination and checks each returned batch to see if it's a full batch, then if it is it requests the next batch presuming there are more matches. With HTTP/2 push it might be better to ask the server to push all matches to us since the server knows when there are no more matches.

[jmakeig]

With the upcoming HTTP/2…

There are no immediate plans to implement HTTP/2 in MarkLogic’s HTTP server. However, it’s something we will be assessing for a future release. The hypothesis is the HTTP/2 will provide better performance and conformance, especially for workloads like the Java Client API.

[sammefford]

Thank you, Justin, for clarifying.

[sammefford]

I've done a lot of evaluation of many HTTP libraries. The most popular libraries: Apache HTTPClient and URLConnection are both in "alpha" or "incubator" status on complete overhauls due to two upcoming changes: HTTP/2, and reactive-style (async) development. While either choice has current stable releases that would serve well our current needs, we'd most likely want to throw them out before too long to take advantage of their new incarnations.

OkHttp appears to be mature, stable, fast, supported, and it appears it will support all our non-trivial requirements, which include:

• auth types: basic, digest, Kerberos, Certificate-based auth (SSL client cert)
• multipart attachments
• support all REST request types (including PATCH, DELETE, etc.)
• SSL
• pool/reuse HTTP connections
• chunked encoding (streaming) requests
• retry when appropriate
• gzip
• 100/continue

The current plan is to implement #65 using OkHttp. That's slated to happen in 4.0.3.