$data

[handrews]

Originally written by @geraintluff at https://github.com/json-schema/json-schema/wiki/%24data-(v5-proposal)

NOTE: JSON Relative Pointer is defined as an extension of JSON Pointer, which means that an absolute JSON pointer is legal anywhere that a relative pointer is mentioned (but not vice versa).

Absolute JSON Pointers always begin with /, while relative JSON pointers always begin with a digit. Resolving a pointer beginning with / behaves the same whether it is being resolved "relative" to a specific location or not, just as resolving a URI "/foo/bar" is resolved the same whether there is an existing path component to the URI or not.

Proposed keywords

• $data

This keyword would be available:

• inside any schema
• contained in an object ({"$data": ...}) for the following schema properties:
  • minimum/maximum
  • exclusiveMinimum/exclusiveMaximum
  • minItems/maxItems,
  • enum
  • more...
• contained in an object ({"$data": ...}) for the following LDO properties:
  • href
  • rel
  • title
  • mediaType
  • more...

Purpose

This keyword would allow schemas to use values from the data, specified using Relative JSON Pointers.

This allows more complex behaviour, including interaction between different parts of the data.

When used inside LDOs, this allows extraction of many more link attributes/parameters from the data.

Values

Wherever it is used, the value of $data is a Relative JSON Pointer.

Behaviour

If the $data keyword is defined in a schema, then before any further processing of the schema:

• The value of $data is interpreted as a Relative JSON Pointer.
• The pointer is resolved relative to the current instance being validated/processed/etc.
• The resolved value is taken to be the value of the schema for all further processing.

When used in one of the permitted schema/LDO properties, then before any further processing of the schema/LDO:

• The value of $data is interpreted as Relative JSON Pointer.
• The pointer is resolved relative to the current instance being validated/processed/etc.
• The resolved value is substituted as the property value.

Example

{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "type": "object",
    "properties": {
        "smaller": {"type": "number"},
        "larger": {
            "type": "number",
            "minimum": {"$data": "1/smaller"},
            "exclusiveMinimum": true
        }
    },
    "required": ["larger", "smaller"]
}

In the above example, the "larger" property must be strictly greater than the "smaller" property.

Concerns

Theoretical purity

Currently, validation is "context-free", meaning that one part of the data has minimal effect on the validation of another part. This has an effect on things like referencing sub-schemas. Changing this is a big issue, and should not be done lightly.

Some interplay of different parts of the data can currently be specified using oneOf (and the proposed switch) - but crucially, these constraints are specified in the schema for a common parent node, meaning that sub-schema referencing is still simple.

The use of $data also (in some cases) limits the amount of static analysis that can be done on schemas, because their behaviour becomes much more data-dependent. However, the expressive power it opens up is quite substantial.

Not available for all keywords

It's also tempting to allow its use for all schema keywords - however, not only is that a bad idea for keywords such as properties/id, but it also might present an obstacle to anybody extending the standard.

Not available inside enum values

It should be noted that while {"enum": {"$data":...}} would extract a list of possible values from the data, {"enum": [{"$data":...}]} would not - it would in fact specify that there is only one valid value: {"$data":...}.

Similar concerns would exist with an extra keyword like constant - what if you want the constant value to be a literal {"$data":...}? However, perhaps constant could be given this data-templating ability, and if you want a literal {"$data":...}, then you can still use enum.

Describing using the meta-schema

The existing mechanics of $ref can be nicely described using a rel="full" link relation.

The mechanics of $data, however, would be impossible to even approach in the meta-schema. We could describe the syntax, but nothing more. Is this a problem?

[handrews]

I feel that there are several use cases here, and it might be best to split them up.

URI Template resolution

For hypermedia interactions, where the instance data must be referenced in order for hyperlinking to work at all, the extended templating syntax from issue #52 covers the necessary use cases. It narrowly targets hyperlinking and does not involve instance data in any other aspect of JSON Schema, as it applies only in situation where we were already referencing instance data.

Link title

I can see how a link title might reference data in the same way as the URI Template. The template may include the id of a related thing, while the title may include the related thing's name. Either way, the URI and the title are both things that are presented back to the user, which should be affected by the instance data as they are describing a relation involving that data.

Therefore I would prefer to see the URI Template extended syntax (with "vars") be used here rather than a more generic approach that applies to more than just hypermedia values.

rel and mediaType

The use cases for mediaType and rel are not immediately obvious to me. The relation type should not, in my mind, change based on the instance data. Only the specific instance to which the relation points should change. A mediaType specified at runtime from instance data would not be of use in planning what a program can do with different representations. It's not clear to me why you wouldn't just list out the possibilities. If some media type links may only be present some of the time, there are other ways to express that using "oneOf" (or possibly "switch") to associate links with only certain variations of the content.

Interactions during validation

I feel like this should somehow express the constraints in terms of the relations among the fields rather than pulling in data that will produce the desired result. So somehow explicitly saying that "larger" should be strictly greater than "smaller" without loading the data into the schema before validation.

I'm not 100% sure what that would look like. In this case, possibly very much like the $data example, as either way you need to reference the related field. But I would be more comfortable with something that clearly states "this is describing relationships among data" rather than "this loads a value from instance data, and treats it as part of the schema, whatever that happens to mean."

I don't feel like I'm articulating this well, but I'm going to go ahead and post this comment in the interest of provoking discussion :-)

[HotelDon]

@handrews
So, for me, my use cases sit entirely in the interactions during validation category, so that's what I'm going to speak to.

I think I have a basic understanding of what you're trying to say - you'd prefer a solution where JSON Schema validators use $data (or some other similar feature) as a pointer whose value is checked during validation, instead of recompiling the schema before validation even begins, where it inserts the value of those pointers directly into the schema.

Would it be possible to modify the proposal to remove the portions about modifying the schema directly, and include language elsewhere in v6+ that validation shouldn't modify schemas for any reason? Then, $data would be functionally identical to the way it works now, but doesn't encourage "bad behavior" among the various JSON Schema validators.

[handrews]

@HotelDon this is what I mean by "I'm not articulating this well"

It's not so much the reading/loading of the data (which likely has to be done lazily because of $refs), it's what you can do with it after it is loaded. Although thinking about this more I may be OK with it.

The way the proposal is written, with a list of allowed properties that trails off with "more..." left me very concerned about the scope. However what's not explicitly called out, but I think would be better than listing the fields, is that all of the fields proposed for $data take a literal value, and not a schema. We really need to make sure that data is never interpreted as a schema- that's a security nightmare- just use it to shove in links to all of your favorite malware sites!

But I think the intent here is that $data is only used to load data in place of a literal value. I can get behind that.

I still think it is valuable to separate the hypermedia template resolution cases out and use vars as specified in issue #52 for that. Since the values in vars are already assumed to be pointers into the instances, so requiring them to be little{"$data": "/pointer"} objects instead of just pointers is overkill.

[HotelDon]

I had never considered the possibility of someone trying to load schema's with $data, so I guess that is what got me confused.

So maybe fix it to sound more like this:

This keyword would be available:

• inside any schema
• contained in an object ({"$data": ...}) for most schema properties that accept literal values. For example:
  • minimum/maximum
  • minItems/maxItems
  • pattern
  • enum
  • etc...

I'm still having a hard time wrapping my head around the hypermedia/LDO portion of this proposal, so I don't have much of an opinion on it. It might be helpful if @geraintluff chimed in to defend his original proposal a bit, assuming he still has any interest in doing so.

[epoberezkin]

My 2¢: people seem to use it a lot with Ajv, judging by the questions. So it must be useful.

I think relative JSON pointer should be extended to allow navigating array items (see #115)

[handrews]

I've become more receptive of this proposal while working with some of the more difficult hyper-schema problems such as discussed in #108

[awwright]

I'm solidly of the opinion that checking data consistency is solidly out of the scope of JSON Schema. Although it's certainly an option for validators that do want to offer the feature.

And if it's a popular feature then... maybe it's something we have to look into, perhaps as a separate document though.

[handrews]

@awwright $data has important uses in hyper-schema whether it is available in general validation or not.

[Relequestual]

I can see this could be useful. A few clear usecases might be helpful if anyone has the time or inclination.