Finish and publish Draft 06

[handrews]

Hi folks,
This brings my v6 scope notes out of the depths of the issue with all the comments and into its own tracking issue. I have expanded it with various issues and states at @Relequestual's request. @Julian this is what I referenced in the issue on the test repo the other day.

I'm going to treat this issue a little differently: We will no doubt have discussions in the comments, but I will keep updating this top comment with the current status as we work through things.

TL;DR Highlights

PRs for Draft 06 in priority order

meta-schema updates: #194 or #188, #189 or #185, #190

#179 This hyper-schema issue is the big unknown of Draft 06. It addresses problems with "method", "schema" and "href" that really need to be at least partially addressed for Draft 06.

#171 Wording for when keywords are absent, currently controversial. Touches a lot of language so let's get it resolved ASAP. (this is getting very close to resolution - yay!)

#154 "id" to "$id" is just waiting for @awwright's approval or feedback. I am interpreting @epoberezkin's "thumbs up" on my "let's get this in Draft 06 and keep looking at other issues after that" as approval.

POSSIBLY IN scope for v6

#173 The behavior of "deprecated", and for that matter the conceptual model for resources vs general instances in hyper-schema, is not clear enough to put this in Draft 06. However, if this can be resolved in time, it would be nice to include it.

OUT of scope for v6

#129 Further work past #159, but may take a different form. Ignore for now.

#163 Idea presented for comment, but almost certainly rejected.

Full Details

For most of the open PRs I noted what feedback, if any, the four currently most active people had provided (@awwright, @Relequestual, @epoberezkin, and myself), plus anyone else who has open concerns.

I added a line for tests where appropriate but have not looked into that at all yet. Obviously the new features need tests, and probably some of the clarified things could use regression tests as well.

NEW FEATURES

• Allow boolean schemas everywhere

  • Issue Allow true and false for all schemas (except maybe the root schema) #101 @handrews RESOLVED
  • PR Allow boolean schemas everywhere (root and subschemas) #167 @handrews / @awwright MERGED
  • Tests?

• Change "id" to "$id"

  • Issue Rename "id" to "$id" #20 @JanTvrdik
  • PR Change "id" to "$id", retain "id" as deprecated. #154 @handrews pending
    • @awwright: Considering whether to drop "id" rather than deprecate
    • @Relequestual: Approved
    • @epoberezkin: Gave a thumbsup to my "let's still put this in v6"
  • Tests?

• Remove "method", use "hrefVars" for user input to URI Templates

  • Issue Submitting both user and instance data via both the uri and message body #108 @handrews open # Only addresses part of it, remainder to v7+
  • PR Remove "method", URI input with "hrefVars" #159 @handrews discussion
    • @awwright: disagrees with default variable behavior (could use more opinions?)
    • @Relequestual: agrees with default variable
  • PR Remove "method", URI input with "hrefSchema" (object version) #179 @handrews discussion # More consitent/powerfule alternative to Remove "method", URI input with "hrefVars" #159
    • (just filed, unsurprisingly no responses yet)
  • Tests?

• Add "propertyNames"

  • Issue v6 validation: propertyNames #70 @epoberezkin open
  • PR Validation: Add "propertyNames" keyword #172 @awwright discussion
    • @Relequestual [no response]
    • @handrews requested meta-schema change, concerned about Validation: Replace "considered present" language #171-related language
    • @epoberezkin Approved
  • Tests?

• Add "uritemplate" and "jsonpointer" formats

  • Issue format: add uritemplate and jsonpointer #109 @handrews open
  • PR "uritemplate" and "jsonpointer" formats #150 @handrews MERGED
  • Tests?

• Numeric "exclusiveMinimum" / "exclusiveMaximum"

  • Issue Keywords for exclusive minimum/maximum #77 @handrews open
  • PR n/a @awwright direct commit # specification
  • PR Validation: Add note about removal of boolean forms of exclusiveMinin… #162 @awwright MERGED # deprecation note
  • PR Add "$ref", numeric exclusive*, default empty string arrays to meta-schemas, standardize meta-schema formats #168 @handrews open # meta-schema
  • Tests?

• "examples"

  • Issue example attribute #121 @geemus RESOLVED
  • PR Add the "example" keyword. #123 @handrews MERGED # specification
  • PR Add "$ref", numeric exclusive*, default empty string arrays to meta-schemas, standardize meta-schema formats #168 @handrews open # meta-schema
  • PR Add release note for "examples" #169 @handrews MERGED # release note
  • Tests?

• "contains"

  • Issue Add contains keyword #32 @BigBlueHat RESOLVED (I accidentally dup'd this in v6 validation: "contains" #63)
  • Issue v6 validation: "contains" #63 @geraintluff RESOLVED (part rejected)
  • PR Add the "contains" keyword for array instances. #113 @handrews MERGED # specification/meta-schema
  • PR n/a @awwright direct commit # release note
  • Tests?

• "const"

  • Issue v6 validation: "constant" #58 @geraintluff RESOLVED
  • PR Add "const" keyword for constant values. #139 @handrews MERGED # specificatino/meta-schema
  • PR n/a @awwright direct commit # release note
  • Tests?

• Allow empty "required" and "dependencies" string arrays

  • Issue Allow empty "required" array #69 @gibson042 RESOLVED
  • PR Allow an empty array for "required". #112 @handrews MERGED # specification / meta-schema validation
  • PR Add "$ref", numeric exclusive*, default empty string arrays to meta-schemas, standardize meta-schema formats #168 @handrews open # meta-schema default
  • PR n/a @awwright direct commit # release note
  • Tests?

• meta-schema update for v5 changes

  • Issue n/a
  • PR Update meta-schemas for changes in the last draft. #132 @handrews MERGED # v5
  • PR Add "$ref", numeric exclusive*, default empty string arrays to meta-schemas, standardize meta-schema formats #168 @handrews open # "$ref"
  • Tests?

CLARIFICATIONS

• "may be considered" language for default values

  • Issue "may be considered" #131 @joe-chung closed
  • PR Validation: Replace "considered present" language #171 @awwright discussion
    • @Relequestual suggested alternate wording
    • @handrews objects to change (may be OK with different wording)
    • @epoberezkin [no response]

• "additionalItems" is ignored when "items" is not present, because by default "items" can be considered present with an empty schema.

  • Issue "additionalItems" should not be honored if "items" is an object. #135 (comments) RESOLVED
  • PR 156 @handrews MERGED
  • Tests?

• Make "items"/"additionalItems" wording consistent with *properties

  • [no real issue for this, got done while addressing issue additionalProperties behavior for non-object instances #103]
  • PR Make properties/items wording consistent. #111 @handrews MERGED

• General validation principles

  • Issue v5 validation: Clearly document validation principles #55 @handrews open # dropped other stuff in v5 validation: Clearly document validation principles #55
  • PR Add general validation principles and examples. #143 @handrews discussion # made changes, awaiting feedback
    • @awwright: Requested changes, no response since they were made
    • @Relequestual: [no response]
    • @epoberezkin: thumbsup on original PR comment

• A note about current meta-schema URIs

  • Issue Publication of meta-schema #165 @awwright open
  • PR Add note about current meta-schema URIs #164 @awwright MERGED
    • @Relequestual: Approved
    • @handrews: Objects to the use of "draft-04" on master
    • @epoberezkin: [no response]

• Mention of utility with CBOR and other media types

  • Issue Investigate CBOR compatibility #6 @awwright open
  • PR Core: Mention alternate media types like CBOR #153 @awwright MERGED

BUG FIXES

• "integer" type removed?

  • Issue "integer" type removed? #146 @handrews RESOLVED
  • PR Correct number of primitive types from seven to six. #145 @kw317 closed (this removed it entirely)
  • PR Validation: Fix reference to seven primitive types #147 @awwright MERGED
  • Tests?

• "additionalProperties" fails validation of non-objects

  • Issue additionalProperties behavior for non-object instances #103 @handrews RESOLVED
  • PR Make properties/items wording consistent. #111 @handrews MERGED
  • Tests?

• hyper-schema examples were wrong

  • Issue Hyper-Schema examples: wrong, confusing, or changed? #97 @jdesrosiers RESOLVED
  • PR Fix parent link example. #138 @handrews MERGED

• hyperschema references deleted example

  • Issue spec: 3rd link in example removed but still referenced #91 @handrews RESOLVED
  • PR n/a @awwright direct commit

• various typos

  • Issue n/a
  • PR A couple of typos #166 @bowman MERGED

[epoberezkin]

@handrews great summary, thank you. I will have a look over weekend.

[awwright]

I've been thinking it would be prudent to get as many new keywords in as possible, and any other meta-schema changes in, before publishing a new I-D.

I'm trying to find any other issues to tag under https://github.com/json-schema-org/json-schema-spec/milestone/2

I've pushed other issues out to a new "future" milestone. This includes one or two Hyper-schema meta-schema changes that should wait on other stuff first.

This list seems a little sparse though, if there's anything that's missing let me know.

[epoberezkin]

@awwright @handrews my only concern in that list is about #76. I thought the consensus was to do nothing about it and leave it as is. Because otherwise draft 6 will not be backward compatible and my understanding was that we want to have somewhat extended, clean and backward compatible I-D (as the previous one was a bit rushed-out). Am I wrong? Or is this list about draft-7?

[handrews]

@epoberezkin what about #76? It is not even clear what should be done from that one as there's not a clear push for "additionalPatternProperties" and there's no support for other things raised in that issue.

[handrews]

@awwright you're missing the most obvious one #159 and also #109 and #143 .

The point of the earlier discussion in #137 was to get Draft 06 out quickly due to the issues that the rest of us have with Draft 05. I don't want to get Draft 06 bogged down in a frantic push to shove as many features as possible into it. Isn't the point to do frequent drafts for feedback? It's better to get each successive draft out the door than to hold them as long as possible.

[handrews]

@awwright having considered this more and taken a look at your recent PRs I have some concerns:

1. If you wanted to expand the scope of Draft 06, you could have easily spoken up in Need a contributor's guide #137 last week after I posted the earlier version of this and @Relequestual and @epoberezkin both gave feedback. But instead of responding to our efforts to, among other things, accommodate your concerns about the pace of draft releases, you just expanded the work with no discussion.

2. While "propertyNames" is probably fine to add to Draft 06 (I'd had it in the list until @epoberezkin asked for a fast rather than large release), "deprecated" is not ready and has not had enough discussion. Exactly how it fits into hyper-schema's notions of resources (already a bit unclear) vs fields is not obvious and we should not be holding up Draft 06 for it. Of course if we resolve it in the next day or so we can include it, but that seems unlikely right now.

3. Please work on resolving the various open issues before expanding the scope of Draft 06. Most of them are blocked awaiting your approval or feedback, in some cases for multiple weeks with no acknowledgement from you. It has been very frustrating to watch you push back on getting reviews because you think it will slow us down too much, but then ignore PRs for days or weeks and add more work to Draft 06 when the rest of us are trying to wrap it up.

4. @Relequestual discussed how he wanted to manage the issues and PRs here in Need a contributor's guide #137 and has started making changes to the tags in the various repos. But you have now come up with an entirely different approach to managing things are are just doing that without discussion. And not quite aligning with what is here either, as noted in an earlier comment. We need to agree on a way to manage the draft process. Particularly when there is already a robust discussion among the rest of the community, you should not just run off and do something totally different. As with direct commits, when you take unilateral action on something that is under discussion within the community, you are showing a significant lack of respect for this community.

[handrews]

@awwright @Relequestual @epoberezkin I have updated the initial comment with a rough priority order for resolving the open PRs, focusing on getting the ones with the most uncertainty and/or broadest impact resolved quickly.

[epoberezkin]

what about #76? It is not even clear what should be done from that one as there's not a clear push for "additionalPatternProperties" and there's no support for other things raised in that issue.

@handrews exactly. I thought we would just drop it. I meant that I am concerned that it is added to the milestone (@awwright wrote: I'm trying to find any other issues to tag under https://github.com/json-schema-org/json-schema-spec/milestone/2).

[handrews]

@epoberezkin I have no idea what @awwright is doing with his milestones since he has not coordinated with what you, @Relequestual and I discussed in #137. As far as I am concerned, #76 is not in Draft 06 and we should not be hunting for more work. We should be finishing the open work that we already have in flight and getting a draft out.

Until at least the four of us can come to an agreement here, which I would like, on how we are managing this release, I am going to continue on the path that we discussed before. There are many ways that I would be perfectly happy to see the release managed, but we do need to agree on a single approach.

[awwright]

@epoberezkin I'm guessing #76 will be closed without making any changes, but it'd probably be a good idea to address the author's use case somehow. I'll continue discussion there.

Edit: Actually if that's the case I can just de-prioritize it by removing it from the "milestone"

[awwright]

@handrews What do you actually need from this issue?

If you'd like an I-D published, I can do that now, but I'd like to suggest holding off until most of the new/renamed keywords have been merged in. Maybe another week or two.

If you're waiting for a new feature to be merged in first, you can let me know in the respective issue.

[handrews]

@awwright this issue is defining the scope and tracking the progress for Draft 06.

Obviously I do not want a draft published right now. There is whole section of "PRs that need to be resolved" which, as you might guess, need to be resolved first. But I also do not want to keep tacking more and more stuff on. This is a very standard scoping exercise, is it really that hard to see?

I'm going to refer back to #137: You have repeatedly ignored the requests from the community for better project management. In response, you give your personal preferences, which we have all indicated we feel are inadequate to run the project. Yet you keep acting however you like instead of trying to work with us.

So I proposed something for scope and focus. I got feedback from @Relequestual and @epoberezkin and made some adjustments. @Relequestual suggested giving me write access to the repo so I could manage the issues and milestones, which apparently you are vetoing by just ignoring the suggestion entirely- I'm going to leave that decision up to the two of you and not discuss it further here.

Without write access, and therefore without the ability to manage milestones and labels, I'm working here in an issue to follow on the path that the three of us discussed. @Relequestual asked me to collect all of the PRs and issues involved in Draft 06, so I did that here.

But as you so often have, you just ignore what the rest of it do when you don't care for it, and tell us to do things your way. Which you don't explain very thoroughly, and change whenever it suits you.

If you want to facilitate JSON Schema in terms of project management, then you need to communicate clearly, consistently, and frequently. You need to actually listen to the rest of us and incorporate our feedback instead of just repeating the same explanations of why you're doing whatever you want. You should be able to tel by now that despite respecting your technical abilities, when it comes to the process here we're not buying your approach.

Either work to get us on board with how you plan to project manage (which is still not at all clear to me), or get involved in our discussions and work within the framework that we're attempting to produce through collaboration. Or at minimum, stop working at active cross-purposes.

I'm only attempting to do this because you have not. I don't need to be the project manager, but I have more time to devote to this project than (as far as I can tell) anyone else, and I need it to be managed by someone. I'm not claiming I have the best approach, but I have a clearly communicated approach that has gotten support from other active community members. If you have concerns about that, we can change that, but you need to convince us and get clear support from us rather than just doing what you've been doing.

[awwright]

@handrews I'm sorry if progress forward isn't going as fast as you like. I'm actively trying to moderate that and other expectations, even when it's not the most popular thing to do.

The IETF is very clear on what an Internet-Draft may be used for, and those are the constraints I have to follow as document editor, it's the same process used by all the other groups doing the same thing. Anguish with this process is not unfamiliar to me, I understand! I've literally been laughed at for suggesting some of the things you have.

Overall, I'm trying to accommodate as many requests as possible, but if there's a lot coming in, that necessarily means that issues have to languish. My apologies if that looks like I'm ignoring them!

Right now my plans for I-D are to wait until most of the features in https://github.com/json-schema-org/json-schema-spec/milestone/2 are checked off. If you would like to see something different, by all means let me know, but this is a rather verbose issue and I'm not exactly sure what changes it's proposing.

I also want to note you're posting very lengthy issues in general that I'm doing my best to work with. I want to keep issues on-topic, please drop me an email or meet up in IRC.

[handrews]

I'm sorry if progress forward isn't going as fast as you like.

@awwright it has nothing to do with speed and everything to do with you not listening, as you just demonstrated with this response.

My problem is that you make this all about you. It's all about what you will do when you will do it, and what you have decided is what the project will do. It's never about "us" (the community, not you and me specifically). @Relequestual and @epoberezkin have been much more collaborative, and that is even with @epoberezkin and I vehemently disagreeing on technical points much of the time. I feel like the three of us are all trying to move together in roughly the same direction (although I invite either of the other two to object if I am mischaracterizing their views).

You, on the other hand, are doing your own thing and frequently surprising everyone with decisions out of the blue, or duplicated work, or work that crosses other work going on.

If you want to be "the leader" (in quotes because theoretically this is a community project and not owned by a single leader), it is your responsibility to stay on top of the project. This issue is specifically for managing the large incoming workload and backlog, so when you say you're trying to balance that but at the same time ignoring all efforts by anyone else to bring clarity and focus to that exact problem, I have no sympathy for your position. If you do not have time to keep on top of everything and manage expectations, then let someone else do it. The document editor/technical lead and the project manager don't need to be the same person. In most organizations they are not.

[awwright]

@handrews I'm afraid there's indeed been a lot of times I have to inform everyone it's not possible for us to do things a certain way. To that end, I try very hard to separate my personal views from my role in editing the document.

Please see me on IRC if you want to discuss specifics, issues and other long-form responses tend to be exceptionally poor at pegging down specific misunderstandings.