LDO "get" method is too ambiguous

[jdesrosiers]

I'm just getting around to reviewing the new Hyper-Schema spec and I have a few issues. Sorry, it's a bit late. I'm doing the best I can to keep up with the lightning fast pace this project suddenly started moving at.

If "method" is "get", the link identifies how a user can compute the
URI of an arbitrary resource. For example, how compute a link to a
page of search results relating to the instance, for a user-selected
query term. Despite being named after GET, there is no constraint on
the method or protocol used to interact with the remote resource.

The problem I have is the ambiguity. Just like HTML is unambiguous enough for a web browser to generate a UI from HTML, a Hyper-Schema client should be capable of generating a UI (graphical or otherwise) from a hyper-json (JSON Document + Hyper-Schema). Based on this wording, it seems like you can do almost anything you want. What is a generic client supposed to generate when it encounters this LDO?

This is a preserving-the-spirit-of-hyper-schema level issue for me. If I'm missing something, please explain and lets put a better explanation into the next draft of the spec. If I haven't misunderstood, this change is a deal-breaker. I can't use Hyper-Schema like this without assigning semantics to things that aren't in the spec.

[jdesrosiers]

Same thing goes for "post", although it's not quite as bad. The only time you send a request body to the server and it is ambiguous what you should be sending, is when you are doing a POST operation. So, "post" should imply a POST operation. But, there are too many hoops you have to jump through to get to that conclusion.

If "method" is "post", the link specifies how a user can construct a
document to submit to the link target for evaluation.

[handrews]

I just attempted to do something about this and went down in flames (the issues I opened are already closed and are not recommended reading). I appreciate @awwright 's patience although I still share a lot of your concerns, @jdesrosiers .

My current understanding is this:

• All of this terminology is by analogy with HTML forms, and therefore incorporates how the HTML form element behavior has evolved as the web has expanded beyond HTTP/1.0, even though HTML5 still has pretty much the same form syntax.
• A link that takes input is a form. There's nothing more to it than that.
• A "get" link/form encodes its input in the URI. That's all that "get" means. It has nothing to do with HTTP GET or any sort of more abstracted retrieval operation. It does not mean that sending input to the form is safe.
• A "post" link/form encodes its input in the request body. That's all that "post" means. It has nothing to do with whether sending input to the form involves safe, unsafe, idempotent, or non-idempotent behavior, nor is it related to the protocol method used.

---

I'm more or less fine with the idea that there only links that do or don't take input. I'm still working through the details, but this is why I like JSON Home as a complement to JSON Hyper-Schema, because it is oriented towards hinting at the semantics and mechanisms that Hyper-Schema is (apparently) intentionally avoiding.

I somewhat dislike calling links that take input "forms". I am particularly confused that "form" is considered to be good terminology here, but the "create-form" and "edit-form" links are considered invalid to use. This means that we're using "form" for different things, and I do not really understand why. The link rels, I'm told, are only for interactive forms, but I can't figure out how the RFC for them imposes any such constraint.

I very strongly dislike both the use of the terms "get" and "post" and their use to separate encoding of input into the URI vs the request body.

You've already observed the horribly misleading nature of these terms given the specified usage. So have I, including an extensive example of why I think it is a problem for adopting Hyper-Schema because it results in very poor UX for schema designers (I can elaborate on this concept of UX if needed).

@awwright 's response was:

We're designing a specification whose audience is developers of validation libraries. So when I say that "get" and "post" are opaque tokens, that's because we can make the vocabulary more intuitive later, right now we're more concerned with how is this going to work.

Needles to say, I disagree with this- the time to fix vocabulary is now, before too many people adopt (and likely misuse compared to the current specification) the current vocabulary.

As far as encoding:

• We have far more powerful tools available with URI templates
• It should be possible to resolve data from both the instance and user input whether it ends up in the URI or a request body

I'm not sure why we're preserving this split explicitly when it is rather specific to HTML's usage of HTTP/1.0

[handrews]

I forgot to elaborate on the "how the HTML form element behavior has evolved" comment. In the epic "does anyone actually use hyperschema?" issue, @awwright said:

Even in HTML, method="get" and method="post" doesn't necessarily mean an HTTP operation using that method. Indeed, <form method="post" action="mailto:robot@example.com"> ... is supposed to send an email, not make an HTTP request.

... and I've been trying to work with that since. But I feel a bit better about my confusion, because W3C's HTML5 specification still defines method as:

method - HTTP method to use for form submission

You have to dig down into the algorithm for form-submission to find this part:

Otherwise, select the appropriate row in the table below based on the value of scheme as given by the first cell of each row. Then, select the appropriate cell on that row based on the value of method as given in the first cell of each column. Then, jump to the steps named in that cell and defined below the table.

            GET                 POST
http        Mutate action URL   Submit as entity body
https       Mutate action URL   Submit as entity body
ftp         Get action URL      Get action URL
javascript  Get action URL      Get action URL
data        Get action URL      Post to data:
mailto      Mail with headers   Mail as body

If scheme is not one of those listed in this table, then the behavior is not defined by this specification. User agents should, in the absence of another specification defining this, act in a manner analogous to that defined in this specification for similar schemes.

[... much more detail about the stuff in the table ...]

It's not clear to me why we should model ourselves after something that's basically a switch statement on protocol. As far as I can tell HTML having something like this is an artifact of it originally being designed for HTTP/1.0 and needing to map that design into other protocols. We can make things a lot better by designing something that is not constrained by HTTP/1.0 history.

Some of these have no distinct "post" behavior. HTTP and email have separate actions, but while the "body" concepts are reasonably analogous, email headers and URL query string parameters do not necessarily have the same or even similar semantics.

We should have one way to define what input can be accepted and where to put that input, whether that is through URI templating or message body construction. The "where" should be determined by either mapping the input to URI template variables (in which case it goes in the URI, more flexibly than with form query encoding), or by not mapping it (in which case it goes somewhere else which is probably protocol-dependent as specified by the URI scheme, but not constrained by Hyper-Schema).

That input system should also optionally map instance data into the input (no matter whether the input is for the URI or not). Some input may not be resolvable from the instance, and some URI template variables may not be available for user input, so we should be able to express those things as well.

[handrews]

I just went and looked at draft 04's meta-hyper-schema:

"method": {
    "description": "method for requesting the target of the link (e.g. for HTTP this might be \"GET\" or \"DELETE\")",
    "type": "string"
},

and spec:

This property defines which method can be used to access the target resource. In an HTTP environment, this might be "GET" or "POST" (or other HTTP methods).

Some link relation values imply a set of appropriate HTTP methods to be used for the link. For example, a client might assume that a link with a relation of "edit" can be used in conjuction with the "PUT" HTTP method. If the client does not know which methods might be appropriate, then this SHOULD default to "GET".

So I am not losing my mind, and when I read Hyper-Schema years ago, I did not fail to comprehend how it used this keyword. Specifying full operations by HTTP method was actively encouraged. The things we did at Riverbed followed directly from this definition. It may never have been the right way to think about links, but it was what was advised at the time here.

That means this new wording is actually a major change. (And I feel a lot better about my mental capacity.)

[jdesrosiers]

Thanks, @handrews, that was extremely helpful. We need to do a much better job at communicating this in the spec. We don't want to duplicate the HTMLforms documentation, but right now it makes it sound like the semantics of "method" are largely undefined.

I have not read the HTMLforms spec, but I think what I am hearing is that at least when HTTP(S) is used, "post" not only means "Submit as entity body", but also means the "HTTP method to use for form submission". This interpretation would be consistent with the way browsers implement forms.

So, it is now my understanding that the spec is just saying that when "method" is used, an LDO uses the same semantics as an HTML form. So, this doesn't really change the semantics of "get" and "post" from what they were in draft-04 when using HTTP. @awwright, can you confirm that I understand this correctly?

Am I correct that Hyper-Schema has lost the ability to describe that "PUT" and "DELETE" operations are allowed? Is "method" not allowed to be "PUT" or "DELETE" in an HTML form? I know it works for ajax calls. It would be unacceptable to have to require clients to make an OPTIONS request to know if "DELETE" and "PUT" are allowed. It's just not practical.

If this is correct, we shouldn't have published the new draft with this functionality removed. The "allow" proposal (#73), could fix the problem. That or something like it should be added ASAP because Hyper-Schema is incomplete without it.

[handrews]

@jdesrosiers You're welcome! My feeling is that JSON Home, either directly or by the example of its "hints" concept with which "allow" fits, is the answer to the OPTIONS problem. I'm mostly sold on the idea that features that non-authoritatively optimize protocol deficiencies should be separate in some way from the main Hyper-Schema features. And also that it should be possible to construct a fully-functional hypermedia system without them, although it may be slow and inconvenient.

[awwright]

@jdesrosiers I believe you understand correctly. The new draft, for the most part, is reverse-compatible, with some changes in language and, where necessary, definitions in order to bring the language up-to-date with the state of the art of hypermedia systems.

One of those lessons is that links don't tell you how to interact with a resource; they tell you the relationship between resources. Once you know a resource exists, then you select how you want to interact with it. If it's an HTTP resource, you select a method like GET to retrieve the resource, or you use PATCH or PUT if you want to change it, DELETE if you want to remove it, etc. There's no need for the link to re-define everything that's already defined in HTTP.

The HTML spec is increasingly becoming the "Web browser specification" and defines a lot of things that really have nothing to do with HTML (it has a whole section on what to do if the document is not HTML, something I believe grossly out of scope).

So when it says "Get action URL", it's talking about what the behavior of the submit button is, not what the form means. So the behavior between HTML and JSON Schema is still fundamentally the same: A method="get" form tells us how to construct a URI for an arbitrary resource (like a search query), and it is something you can PUT to or PATCH to if you're so inclined.

[awwright]

@handrews Is there some difference between HTTP/1.0 and regular HTTP I'm unaware of? HTTP/1.1 is the latest spec.

Also, I've never had to use OPTIONS, so I guess I'm still unsure exactly what problem it is that you're trying to solve. Most all of the time you can just opportunistically make the request you want to make. If the server doesn't support it, it'll come back and tell you so.

[awwright]

@handrews:

Needles to say, I disagree with this- the time to fix vocabulary is now, before too many people adopt (and likely misuse compared to the current specification) the current vocabulary.

The explicit goal of the latest I-D was to be reverse-compatible, without any updates to the meta-schema. That's all.

If there's a better idea for how to say "this relation accepts user input" or "here's how to encode a POST submission", now's the time to suggest it!

[handrews]

The explicit goal of the latest I-D was to be reverse-compatible, without any updates to the meta-schema. That's all.

If there's a better idea for how to say "this relation accepts user input" or "here's how to encode a POST submission", now's the time to suggest it!

The latest I-D has already been submitted when you wrote that, so you'll have to forgive me for assuming that you were blowing off my concerns in general. Perhaps I should have kept using the "v6" prefix that I dropped because you said you didn't like it?

[handrews]

Is there some difference between HTTP/1.0 and regular HTTP I'm unaware of? HTTP/1.1 is the latest spec.

Actually, HTTP/2 is the latest spec, and transparent use of either HTTP/1.1 or HTTP/2 (and possible some non-HTTP protocols) is a top priority of my current REST project.

HTTP/1.0 only specified GET, POST, and HEAD. It was more of a description of what was implemented at the time rather than a specification for implementation. I don't know whether HTTP/1.0 defining only GET and POST was because of HTML, or HTML only allowing methods of "GET" or "POST" was because of HTTP/1.0. The important point is that HTML5 has never moved away from this, and the messy "get"/"post" terminology is the result.

Were you to start HTML over today with HTTP/2 or even HTTP/1.1 as the primary expected protocol and no legacy constraints, no one would design "method" to take "get" and "post" as its two values. They only even sort-of make sense when you fully understand the history.

[handrews]

So when it says "Get action URL", it's talking about what the behavior of the submit button is, not what the form means. So the behavior between HTML and JSON Schema is still fundamentally the same: A method="get" form tells us how to construct a URI for an arbitrary resource (like a search query), and it is something you can PUT to or PATCH to if you're so inclined.

With all due respect (and based on your work and patience here, that's really quite a lot), you are too close to this to see how poorly it is landing on everyone else who has commented so far. [edit: we're all too close to parts of this- see next comment]

Yes, you've sliced the things implied by the HTML spec that tell the browser how to use the data from the part where the browser actually picks an HTTP method for sending the data, but that is not a separation that is clear from the resulting terminology. To be blunt, even though I understand and support the conceptual model you are pushing with this description, I am not buying the description and terminology in the slightest.

These conversations have significantly helped and improved how I reason about specifying links in hyper-schema. So you are generally convincing on this topic. But when it comes to this way of speaking about things, it's about as unconvincing as anything I've ever encountered. Being technically correct doesn't mean it makes any sense.

(and yes, I'm writing a proposal with alternate terminology and a better coverage of the link/form/instance combinations allowed by HTML but not JSON Hyper-Schema, although with considerable trepidation given your tenacity on this point)

[handrews]

@awwright said:

If there's a better idea for how to say ... "here's how to encode a POST submission"

Why are we even talking about "here's how to encode a POST submission"? You're all about not specifying PUT and PATCH, so we wouldn't talk about 'here's how to encode a PATCH submission". We use Accept-Patch for that.

And please don't tell me you don't mean HTTP POST. You capitalized it, so it reads as an actual HTTP method, and not an HTML form method of "post".

If you did mean an HTML lower-case "post", then clearly you're having trouble keeping this straight which should really go a long way towards explaining why @jdesrosiers and I are pushing back on this as confusing or misleading.

If you actually meant HTTP POST, then I am totally lost and want to hear why we are talking about HTTP POST but shouldn't talk about encoding HTTP PATCH. We should either be talking about both (because POST meant "post" which doesn't mean POST) or neither.

[awwright]

Perhaps I should have kept using the "v6" prefix that I dropped because you said you didn't like it?

Well, I said it's going to be disappearing from the IETF documents. I don't really have a better alternative to suggest, though. The milestones are still draft-5 and draft-6.

[awwright]

Actually, HTTP/2 is the latest spec, and transparent use of either HTTP/1.1 or HTTP/2 (and possible some non-HTTP protocols) is a top priority of my current REST project.

HTTP2 is a completely different encapsulation of all the same messages that can be carried by HTTP/1.1. There's few, if any, semantics changes. The methods available for use in HTTP is defined in the IANA registry, the individual methods are spread out across a number of different specs.