pagination example in hyper-schema

[dlax]

In #448, a paginated collection example got added, with pagination handled using hyperschema links. This makes example fairly complicated, especially (I think) because the "thing-collection" schema is an object with metadata to hold pagination information instead of a plain array.

For some context on the initial discussion with @handrews and @philsturgeon, see #448 (comment).

Quoting @handrews #448 (comment)

Collections that are filterable or paginated/scrolling are unmanageable under hyper-schema without meta-data, and showing a "collection" that can't be filtered or paginated is not realistic.

I would don't see a compelling reason to use hyper-schema to handle pagination. In the "thing-collection" is a plain array, prev/next links could be HTTP Link headers I think. Maybe this design is less sophisticated, but I think it does the job.

[handrews]

@dlax I really don't understand why you want this to be a plain array. I just haven't seen that as best practices for collections for a long time. I would consider it unrealistic.

Yes, you can use HTTP Link headers for this, but this is the Hyper-Schema spec so the point is to show how to do things with Hyper-Schema. There's no reason to not use Hyper-Schema here. Having your links split up among Hyper-Schema and Link headers would be excessively confusing.

Duplicating certain links, particularly easily recognizable ones like pagination, in both places would make sense. And the Overview notes that you can use Hyper-Schema on the server side to generate Link headers. But I don't see one person's inclination to not use Hyper-Schema for the most common pattern in REST to be a reason to keep it out of the spec.

For me, the entire point of a hypermedia format is to do your hypermedia in the hypermedia format. So I'm really not finding any aspect of this argument- the array part or the Link header part- to be convincing. I strongly dispute that there needs to be any sort of reason, compelling or otherwise, on a link-by-link basis to use Hyper-Schema rather than Link headers. You choose your hypermedia approach and apply it consistently.

I'm also literally on my way out the door for vacation, so I'll have to get back to this next week when I return.

[philsturgeon]

Hey! I think we are having two conversations in one, and one of them is shrouding the other.

1. Should APIs return a collection as an array or an object
2. What should the JSON HyperSchema example show

API's returning a bare array for collections is something I've been wondering about a lot myself recently. With The security vuln I posted about in #448 being kinda irrelevant these days it certainly seems like it could, but as @handrews points out that's definitely going to cause confusion when your links are split between hyperschema and Link headers.

@dlax Regardless of how you or I might feel about collections being an object or an array, my experience (anecdotal as it is), is for objects to be used.

If arrays to you are the common practice, that would explain why you're pushing for arrays.

The goal here is to provide examples based off of common practice, and not a place to try and push for change in those common practices. In the future we might see bare arrays being used more, but at the time of writing, it really does feel like objects are in 99% of arrays I work with (rightly or wrongly).

[handrews]

@dlax @philsturgeon one thing that I believe strongly about hyper-schema systems is that you must always be able to fulfill the "self" link. So if your representation is paginated through the URI (as it should be if it's paginated at all), then your "self" link needs to be of the current paginated (and/or filtered) page.

That typically means that you must have a meta section of some sort, or a pagination section and a filters or query section. There are so many reasons to carry additional fields in a collection representation- aggregate values across the set being another.

Aside from the fulfill-the-self-link rule, which I do feel pretty strongly about, I don't even think the array vs object thing is about which practice is more common. It's about which better illustrates Hyper-Schema. An array collection that relies exclusively on HTTP Link headers isn't wrong, but it also doesn't illustrate Hyper-Schema well.

[handrews]

Personally, I would recommend that next/prev/first/last get serialized as HTTP headers as well so that this particularly common pattern can be easily recognized and used by non-hyper-schema-aware implementations. But that probably goes somewhere else, either the appendix on APIs or on the web site.

[dlax]

My primary concern is making the collection+item understandable and following the comments I made in #448 and #464, my suggestion was to leave pagination aside until this happen and then go with plain arrays to simplify all the things.

So that's about:

2. What should the JSON HyperSchema example show

This is absolutely not motivated by some kind of "best practice". So while I don't quite buy your arguments on this to reject this idea, I'm very open to any suggestion to make the "collection+item" example understandable.

[handrews]

I guess to me, since there are many use cases that require an object-based collection, and nearly all REST APIs use collections, if I, as a reader, can't understand how to do an object-based collection in hyper-schema then I am not going to use the format.

This was a huge debate point when I tried to sell co-workers on draft-04. How do collections work? We all agreed that we needed collection meta-data next to the collection contents. And then what? A lot of the confusion then had to do with the way method was defined in draft-04, so that's not a concern, but recognizing how to use all of the link types correctly within hyper-schema, not HTTP, is still a concern.

[handrews]

@dlax @philsturgeon any further thoughts here? To summarize my position:

• The example should show using link types within hyper-schema. We can add a note about also using HTTP Link headers (if it's not already there), but this is not a spec about implementing collections via HTTP headers.
• I consider pagination (or filtering, but pagination is more well-defined) to be an essential part of illustrating how to build collections in hyper-schema. In my experience, failing to explain a real-world collection in the spec makes it very likely for the spec to be rejected by potential users. Simple examples are great until you try to do something complex. One reason for such a long and extensive example section is specifically to build up something at a real-world complexity level.
• We should stay with an object because it is increasingly common to see objects with meta-data for collections, and because there's no good way to implement the links with an API representation.

So what needs to be done at this point? Are any of the reasons I list above things to which you object?

[dlax]

* The example should show using link types within hyper-schema. We can add a note about also using HTTP |Link| headers (if it's not already there), but this is not a spec about implementing collections via HTTP headers.

I don't think it's need to add such a note. I suggested using HTTP Link header in reply to a question about how I would implement the pagination pattern, not as a proposal to write this in the specification.

* I consider pagination (or filtering, but pagination is more well-defined) to be an essential part of illustrating how to build collections in hyper-schema. In my experience, failing to explain a real-world collection in the spec makes it very likely for the spec to be rejected by potential users. Simple examples are great until you try to do something complex. One reason for such a long and extensive example section is specifically to build up something at a real-world complexity level.

I agree with that but on the other hand if we fail to provide a comprehensive and real-world example (whatever that means), we are not doing a good job either.

* We should stay with an object because it is increasingly common to see objects with meta-data for collections, and because there's no good way to implement the links with an API representation.

I don't buy this argument, but don't object to it either :) Anyways, I reread the "Collections" and "Pagination" example sections and I still find them hard to follow. There's the last paragraph in "Collections" ("But what if we do not have any "thing"s yet?...") which makes the reader feel that there's something "blurry", not to mention the CREF. The "Pagination" section also has a bunch of CREF that gives the same impression. But that's only my feeling, so if anybody else find these sections understandable enough to get the draft released, that's fine by me.

[philsturgeon]

I dont have anything much new to add. :)