Including `unevaluatedProperties: false` in the meta-schema(s)

[gregsdennis]

Since we've made the decision to disallow unknown keywords, the issue has come up as to whether that should be (or can be) enforced by the meta-schema(s).

This issue is opened to continue the discussion in this thread which was unrelated to the discussion at hand.

[gregsdennis]

There are multiple aspects to this:

1. Do we plan on publishing a new meta-schema each year to coincide for the new spec release, or is the meta-schema "living" as well? (I think some of the arguments in the linked discussion assume it will be "living.")
2. If it's a "living" meta-schema
   1. does unevaluatedProperties: false impact forward-compatability?
   2. does that mean $schema will no longer be versioned?
3. Is there a well-defined, intrinsic link between the meta-schema and the support that an implementation provides; i.e. does adding a keyword to the meta-schema imply that an implementation automatically knows about (even without understanding) that keyword? Is this true about all implementations?

[gregsdennis]

1. Is the meta-schema "living?"

This is a genuine question that I'm asking. I'd like to know what others' assumptions have been so far as it makes an impact on this discussion.

2.i. Does unevaluated: false impact forward-compatability?

For a living meta-schema, it can, but only in one case: the implementation doesn't have access to the new meta-schema and chooses to perform a meta-schema validation (which isn't strictly required) on the schemas it processes.

If a foo keyword is added to a living meta-schema that has unevaluatedProperties: false, an implementation that doesn't support foo can behave in different ways when evaluating a schema with foo in it:

• If the implementation performs a validation of the schema with the meta-schema, and it has access to the new meta-schema, it would see that the validation passes, but it would also be unable to process foo.*
  • Not having unevaluatedProperties: false in the meta-schema wouldn't change this case because the new meta-schema defines foo.
• If the implementation performs a validation of the schema with the meta-schema, but doesn't have access to the new meta-schema, it would fail that validation, citing that foo is unknown.*
  • Not having unevaluatedProperties: false in the meta-schema would change this case because the validation would simply ignore foo's presence. The implementation would then fail to evaluate because it's unable to process foo.
• If the implementation doesn't perform a meta-schema validation and just attempts to process the schema (as mine does**), it would identify foo as an unknown keyword. It doesn't matter whether the implementation has access to the new meta-schema.
  • Not having unevaluatedProperties: false in the meta-schema wouldn't change this case because there's no meta-schema validation occurring.

* These describe @jdesrosiers' scenario. The implementation still errors, but the reason changes.
** My implementation (C#) performs a lot of the validations that the meta-schema would perform during deserialization into my JsonSchema model. If the JSON doesn't fit into strictly-typed my model, it's a safe assumption that a meta-schema validation would fail as well, so I deemed such explicit validation unnecessary.

The only impacted case is when the implementation chooses to perform a meta-schema validation and doesn't have access to the new meta-schema. Furthermore, the result is still an error regardless of whether unevaluatedProperties: false is in the meta-schema.

2.ii. Is $schema no longer versioned?

If the meta-schema is living, then it doesn't really make sense for $schema to be versioned anymore. It could just be https://json-schema.org/schema. When we update the spec document, we update the meta-schema.

But then if we're dating the spec releases and implementations are declaring what they support according to those dates, doesn't it make sense to also date the meta-schemas?

3. Is there a link between the meta-schema and implementation support?

I think the disconnect that Jason and I have had in the linked thread comes down to this question and the difference in how our implementations work. (Questions 1 and 2 can be figured out independently of our specific implementations.)

I believe for Jason's implementation, the answer is "yes," whereas for my implementation, the answer is "no."

For my implementation, adding a new keyword to the meta-schema makes no difference because:

1. I package a hard-coded copy of the meta-schema in the library for easy reference. That cannot change without me releasing a new version of the library.
2. I have explicit C# classes for each keyword. Unless a new type is created and registered, my implementation cannot know about a new keyword, even if the new meta-schema is read in and registered (overwriting the hard-coded one).
3. I would never release a package that has knowledge of a keyword without at least some claim and best effort at implementing it.

As a result, my implementation cannot be in a state where it knows about a keyword but doesn't implement it unless a user explicitly creates and registers a no-op keyword class to do so.

[jdesrosiers]

I'm not entirely sure how you're using the term "living", so that's probably the first place we should make sure we're on the same page.

What I mean by "living" is that there's one version that changes over time in a compatible way. Releases would just be snapshots of the current state of the one version of JSON Schema. That includes the meta-schemas. They are a snapshot of the current state of the stable version of JSON Schema.

To me, whether or not meta-schema $ids include a release (snapshot) identifier has nothing to do with whether it's "living" or not. That's a concerns of how we release snapshots. The spec and meta-schemas are "living" regardless of what choices we make with releases (snapshots).

I'll comment more later, but I think we should make sure we're on the same page with this terminology before I comment further.

[gregsdennis]

I agree with all of that, and, yes, that's what I mean by "living."

However I think that in practice, implementors will want to use those snapshots as a basis for claims of compliance, thus effectively making those snapshots, including the meta-schemas, versions of JSON Schema.

The schema author also cares because they want to be sure any implementation that sees their schema supports it. (Granted, any implementation that doesn't will refuse because it'll see "unknown" keywords.)

[jdesrosiers]

I think that in practice, implementors will want to use those snapshots as a basis for claims of compliance [...]. The schema author also cares because they want to be sure any implementation that sees their schema supports it.

Right. That's exactly what I expect them to be used for.

thus effectively making those snapshots, including the meta-schemas, versions of JSON Schema.

That's where you lost me. I can't tell if this is actually a difference in our thinking, or if we're just using the term "version" differently. Here's an example of how I'm using the term "version". OpenAPI 3.1 is a dialect of 2020-12. I don't consider OpenAPI 3.1 Schema to be a distinct version of JSON Schema. It's a dialect of the 2020-12 version. We've been releasing new versions with every release, but there's no reason that we need to do that anymore if releases are compatible. Each release could be thought of as a dialect of the stable version. I'll pause there to make sure we're on the same page.

("version" is far too ambiguous a term for this concept, so I'm open to suggestions on alternate vocabulary to use for this concept.)

[gregsdennis]

OpenAPI 3.1 is a dialect of 2020-12. I don't consider OpenAPI 3.1 Schema to be a distinct version of JSON Schema. It's a dialect of the 2020-12 version.

The word I'd use for this relationship is "extension." I'm also fine with "dialect" for this.

Each release could be thought of as a dialect of the stable version.

I see where you're coming from, however, I don't think most people are going to see it that way. They're going to see the 2025 release as a new, backward-compatible version of the 2024 release.

From what you're saying, it seems like you consider 2025 to be a mere extension of 2024. I just don't think the majority of users/implementors will see it that way.

[gregsdennis]

Oh! I think I have a way to explain the difference I see.

• Extensions (dialects) can be added independently to some base functionality. This is like my data vocab or OpenAPI3.1 building on 2020-12. Neither extension requires the other.
• Versions build on the previous functionality. While 2025 does technically "extend" 2024, 2026 will then "extend" 2025. Each iteration requires (or includes) all of the previous ones.

We're not releasing a base dialect and then releasing extension dialects that can be independently added. Instead, once we "extend" the base, we call that the new base. This means we're operating in versions.

[jdesrosiers]

once we "extend" the base, we call that the new base.

Good. We're on the same page there.

This means we're operating in versions.

We're still using the term "version" differently, so let me take a step back and try to give a semi-formal definition of the concept I'm thinking of. We don't need to call this a "version", but we need some word for it because it's a critical concept.

Definition: A set of dialects that are all compatible with each other. (Given the definition of "compatible" we've settled on previously)

Because all of these dialects are compatible, it doesn't really matter which dialect (snapshot) you're using. The "version" (or whatever we call it) is what implementers are coding for and schema authors are writing schemas for. The dialect doesn't matter for the evaluation of the schema, the "version" is what matters. Since the "version" is the thing everyone is implementing for and writing schemas for, the meta-schema for every dialect should be compatible with any dialect in that "version", otherwise you can get different behavior depending which dialect you're using.

[gregsdennis]

Because all of these dialects are compatible, it doesn't really matter which dialect (snapshot) you're using. The "version" (or whatever we call it) is what implementers are coding for and schema authors are writing schemas for.

The dialect doesn't matter for the evaluation of the schema, the "version" is what matters.

I'm having a hard time parsing what you're saying (in the whole paragraph). Can you elaborate on this? Previously, you've defined a "dialect" as a set of vocabularies. Is that still the case for this conversation? If so, I don't follow what you're saying at all.

I see the version and the snapshot as the same thing.

---

[really oversimplified example]

Let's suppose next year (2024) we release a simple spec that only defines foo and an associated meta-schema that doesn't use unevaluatedProperties:

{
  "$schema": "https://json-schema.org/meta/schema",
  "$id": "https://json-schema.org/meta/schema",
  "properties": {
    "$schema": { ... },
    "$id": { ... },
    "foo": { ... }
  }
}

Someone publishes an implementation and claims 2024-level compliance and for the user's convenience, they include the meta-schema in the package. They call it impl-v1.

The next year (2025) we add bar to the spec and update the meta-schema accordingly.

{
  "$schema": "https://json-schema.org/meta/schema",
  "$id": "https://json-schema.org/meta/schema",
  "properties": {
    "$schema": { ... },
    "$id": { ... },
    "foo": { ... },
    "bar": { ... }
  }
}

The implementor responds by publishing impl-v2 that claims 2025-level compliance.

In order for a user to have a schema that includes bar, they need to use impl-v2 because impl-v1 will see bar in a schema and reject it as an unknown keyword. But, and here's the kicker, the meta-schema packaged into impl-v1 says that the schema that uses bar is valid.

So impl-v1 now presents contradictory behaviors:

• The meta-schema packaged into impl-v1 says a schema with bar is valid.
• Trying to use a schema with bar results in an error saying the schema is invalid.

This is confusing for a user that tries to use bar but neglects to update to impl-v2.

This scenario can be avoided if the meta-schema includes unevaluatedProperties: false.

// 2024
{
  "$schema": "https://json-schema.org/meta/schema",
  "$id": "https://json-schema.org/meta/schema",
  "properties": {
    "$schema": { ... },
    "$id": { ... },
    "foo": { ... }
  },
  "unevaluatedProperties": false
}

// 2025
{
  "$schema": "https://json-schema.org/meta/schema",
  "$id": "https://json-schema.org/meta/schema",
  "properties": {
    "$schema": { ... },
    "$id": { ... },
    "foo": { ... },
    "bar": { ... }
  },
  "unevaluatedProperties": false
}

Now we get consistent behavior between meta-schema evaluation and schema use.

• The meta-schema packaged in impl-v1 says the schema with bar is invalid.
• Trying to use a schema with bar results in an error saying the schema is invalid.

In both cases here, the user is told that the schema is invalid. They look up bar and find that it's a 2025 keyword, which impl-v1 doesn't support. When they update to use impl-v2, which does support 2025, it all works.

I expect most implementations will include the meta-schemas in their packages. But supposing the implementation doesn't, you then have to provide the meta-schema to the implementation, and we're back at the contradictory case because even with the new meta-schema impl-v1 still doesn't recognize or support bar and throws an error.

Note that the above example doesn't "version" the $id/$schema URI (i.e. the meta-schema is "living").

If we do version the URI (and therefore we're versioning the meta-schema), then I think we get the scenario you're thinking of.

// 2024
{
  "$schema": "https://json-schema.org/2024/meta/schema",
  "$id": "https://json-schema.org/2024/meta/schema",
  "properties": {
    "$schema": { ... },
    "$id": { ... },
    "foo": { ... }
  },
  "unevaluatedProperties": false
}

// 2025
{
  "$schema": "https://json-schema.org/2025/meta/schema",
  "$id": "https://json-schema.org/2025/meta/schema",
  "properties": {
    "$schema": { ... },
    "$id": { ... },
    "foo": { ... },
    "bar": { ... }
  },
  "unevaluatedProperties": false
}

Here, https://json-schema.org/2024/meta/schema wouldn't evaluate a schema that uses bar as valid (and I think you think it should).

But that's still fine. A schema that uses bar isn't a valid 2024 schema, and that's what performing this validation is asking.

I don't think anyone would expect a schema with bar to successfully validate against https://json-schema.org/2024/meta/schema any more than a JS dev would expect async/await to be supported if they're using an ECMAScript 2016 interpreter (it was introduced in 2017).

We're not breaking any compatibility promises by doing this, and we're allowing a better user experience.

[jdesrosiers]

Previously, you've defined a "dialect" as a set of vocabularies.

No, I've argued against that definition because there are many dialects that don't have a concept of vocabularies (examples: MongoDB, pre-3.1 OpenAPI, pre-2019-09 drafts). I've always defined a dialect as any distinct flavor of JSON Schema. Dialects aren't a perfect analogy for snapshots and the analogy clearly isn't helping, so let's drop that and try something else. When I said, "dialect" in that paragraph, I just meant snapshot. I think where we disagree is what a snapshot represents. Which brings me to ...

A schema that uses bar isn't a valid 2024 schema

The way I see it, there's no such thing as a 2024 schema. There's only stable-JSON-Schema (SJS) schemas. Implementations evaluate SJS schemas and use release identifiers such as "2024" to indicate that they support all the feature in the 2024 snapshot of the specification. "2024" only describes what implementations support. It doesn't describe schemas.

This is the big win we get from compatible releases. People don't need to care about versions of JSON Schema anymore. The only thing that matters is the capabilities of the implementation (exactly like ECMAScript).

The 2024 meta-schema describes an SJS schema. So do the 2023 and 2025 schemas. That's why I think every meta-schema release should be compatible, they are all describing the same thing, SJS schemas.

So, from that perspective, let's look at your examples. Given the first example that allows additional properties and a schema with the bar keyword,

• The meta-schema packaged into impl-v1 says a schema with bar is valid to the best of it's knowledge.
• Trying to use a schema with bar results in an error saying the bar keyword is not supported/known.

The meta-schema did it's best to validate that it's a valid SJS schema. It can't know the future, so false positives are possible. That's fine. Meta-schemas aren't expected to be able to catch every possible error. The implementation will let the user know that they used a keyword it doesn't understand, just as expected.

Now the second example where additional properties are not allowed and the schema includes the bar keyword.

• The meta-schema packaged in impl-v1 says the schema with bar is invalid. (Wrong!)
• Trying to use a schema with bar results in an error saying the bar keyword is not supported/known.

Assuming the bar keyword is used correctly, the meta-schema produces a false negative result. This is bad because it is a valid SJS schema, that just happens to use keywords this implementation doesn't support. Although a meta-schema can't catch all errors with a schema, it definitely shouldn't say a schema is invalid when it's not.

Lots of implementations (including mine) use the meta-schema to validate schemas before evaluating them. Those implementations would raise an error telling their users that their valid schemas are invalid. The result is an error either way, but the message is wrong.

If the implementation doesn't use the meta-schema to validate schemas, there's no real problem. In that case, the implementation would encounter bar and raise an error saying it doesn't understand that keyword, as expected.

Let me know if all that helps you understand the perspective I'm viewing this problem from.

[gregsdennis]

Okay. I think I see where our difference of opinion lies.

The meta-schema packaged in impl-v1 says the schema with bar is invalid. (Wrong!)

You're saying that an older copy of the meta-schema should validate (successfully) schemas that use newer features.

Suppose we do this. Then the meta-schema can't catch that unknown keywords are used or that future keywords are used incorrectly.

In other words, without unevaluatedProperties: false:

• bar is allowed, even though it's introduced in the future.
• bar: 4 is allowed, even though when it's introduced it'll need to be a string.
• baz is allowed, even though (with my FutureVision® glasses on) it's never going to be added at all.

That second one is really hard to swallow as "future compatible." In 2024, my schema is

{
  "$schema": "https://json-schema.org/meta/schema",
  "$id": "my-schema",
  "foo": 6
}

Now I want to add the awesome bar feature, but I don't read the requirements right, so I add bar: 4 thinking it works like foo.

{
  "$schema": "https://json-schema.org/meta/schema",
  "$id": "my-schema",
  "foo": 6,
  "bar": 4
}

The meta-schema in impl-v1 says it's valid, but the implementation doesn't recognize/support (there is no practical difference) the keyword. So I update to impl-v2. Now the meta-schema says it's invalid. Why? This is a change from before when I was using impl-v1 - a schema that an earlier meta-schema says is valid is now invalid with the current meta-schema. This breaks compatibility.

If the meta-schemas have unevaluatedProperties: false, then this scenario is avoided. The meta-schema would say that bar is invalid, which aligns with what the implementation supports. As a user, when I update the package, the meta-schema now says that bar is allowed (which also aligns with what the implementation supports), and I can move on with my life.

---

You're saying that having unevaluatedProperties: false means, "the old copy of the meta-schema says bar is invalid, but the new one doesn't, and that means the old meta-schema isn't compatible with new schemas." Who cares? No one is going to expect an old meta-schema to validate new functionality. That's not what future compatibility is. Future compability is being able to use an old meta-schema to validate a new schema that only uses the old functionality. That is, we're not going to change stuff that's already defined; we're not going to break anything.

You're also thinking about this from a JSON Schema (the spec itself) and "compatibility theory" (for lack of a better term) point of view, not the user's point of view. But theory has to give way to practice.

I can promise you that the user is going to prefer consistency between what a meta-schema says is valid and what the implementation supports. If the implementation supports up to 2024 and nothing else, then they'll expect that meta-schema validates up to 2024 and nothing else. They're not going to care that the meta-schema in its current form says the new keyword is valid if the implementation doesn't support it.

[Julian]

(I've only tried to read through the thread briefly once.)

To me, we're a bit too in the weeds I think.

The reasons not to put unevaluatedProperties in the metaschema seem to me to be twofold:

• Because we intend to change our release model and guarantees, and alongside it that might affect this portion of the metaschema if it breaks those new future guarantees
• (I don't think referenced anywhere in the thread) -- because it signals that this is behavior specific to some dialect of JSON Schema when I think the intention was to make this a "global" behavior -- i.e. in all dialects, unknown keywords are expected to cause errors now (I still don't fully agree with this, but I'm arguing 'within' what we agreed to move forward with). Putting this in the metaschema then only "fixes" helping users not use unknown keywords for one dialect (ours) but doesn't help all the other new dialects written which also now need to remember to do this. So not adding it and coming up with some other mechanism is slightly attractive.

I might (or might not) agree with the first point if/when we hash out the guarantees and release process, which it seems like a bunch of the comments are trying to do, but I don't think that should affect adding unevaluatedProperties now today, and going off and having that conversation and potentially removing it if it turns out it indeed conflicts with what we end up with. Today, where we definitely don't have a single spec, I think it's fairly uncontroversially a correct thing to have in there, it's just we might change to something where it isn't. If/when we do I think there are probably quite a few things we'll have to reconsider (probably even things in the metaschema). E.g. -- extending a keyword to take more types or more input than before is in theory something that can be done in a backwards compatible way (as previously such schemas were invalid) -- but not if we have a "restrictive" definition of metaschema compatibility where past versions of metaschemas need to be able to validate future versions of keywords where that thing was invalid. Does this mean we should "neuter" the metaschema to not contain these constraints if we agree to make the guarantees from point 1, and also suspect we may want to make such changes in the future? (Don't know.)

(But yeah the above is just from one read, so if I got it wrong, particularly on the 'con' side, I'm sure I'll hear about it :)

[jdesrosiers]

Today, where we definitely don't have a single spec, I think [including unevaluatedProperties: false is] fairly uncontroversially a correct thing to have in there, it's just we might change to something where it isn't.

I've certainly been thinking of this as what is going to to go into the "single spec" meta-schema. I'm fine with including unevaluatedProperties: false for now as long as we all understand that the subject may need to be revisited later on as we hash out the details of how a single spec will work.

[gregsdennis]

Pertinent to this discussion, a user just created a discussion asking why the meta-schema doesn't catch mistyped keywords. Having unevaluatedProperties: false would satisfy their need.

Just worth mentioning.