Guidance on defining collections to group datasets released as a series or in fragments

[philipashlock]

Data.gov has had the notion of a "collection" that can be used to group multiple datasets that would logically be considered a single dataset, but have been released in separate parts. The most common scenario for this is a series of release over time. In some cases a dataset may by published in monthly or yearly releases, but if the only thing that distinguishes these is date, then they should really be packaged as a single dataset. This also makes browsing simpler - it prevents many similar datasets from crowding out more unique ones. Some datasets might also be published by location, such as data relating to each state being released as a separate file. These should also be grouped together to appear as a single dataset.

Ideally agencies should package these all together as a single file/release before publishing, eg one file that is continuously updated is preferable to separate releases over time, but at the very least there should be a way to define this kind of packaged grouping at the metadata level as is currently the case on data.gov.

The way data.gov handles this is that the collection is essentially treated just as a normal dataset entry but it refers to many child entries. Something similar could be done with the data.json schema, but we would need to establish a convention for defining that parent/child relationship between entries.

Here's a current example of a collection on data.gov

View of the collection "parent" metadata:
http://catalog.data.gov/dataset/tiger-line-shapefile-2010-series-information-file-for-the-2010-census-block-state-ba

View of all its "child" datasets:
http://catalog.data.gov/dataset?collection_package_id=2a8b7f0b-1ae5-453c-ba56-996547266a63

[cew821]

👍 Yes this is totally needed. We have a number of datasets at Energy that we would like to consolidate into a single listing rather than have 100s of entries for each year x state.

It sounds like the parent/child "collection" concept used by data.gov is somewhat different than the entry/distribution of datasets currently used by the schema. Should the guidance direct people to put "children" datasets in the distribution array, or is something else needed?

[dsmorgan77]

Collections are absolutely needed. I could argue either way on the ideal publication path (many of DOT's (current) data customers are states or cities who just aren't interested in downloading the entire Nation's data file and filtering out their information ... we should serve both).

The problems with this are that certain properties will need to filter down to the data file itself. A collection may have a temporal coverage of 1975-present, but an individual file may cover only a single year. A collection may have a geographic coverage of "United States" but a single file may have a geographic coverage of "Alabama." Download URLs will be on a file-by-file basis. Formats might change over time. Data dictionaries may change over time as data elements might be changed.

Clear examples where the collections concept is needed include:

• Toxics Release Inventory (http://catalog.data.gov/dataset?q=%22toxics+release+inventory%22&sort=score+desc%2C+name+asc&ext_location=&ext_bbox=&ext_prev_extent=-139.21874999999997%2C8.754794702435618%2C-61.87499999999999%2C61.77312286453146)
• National Bridge Inventory (http://catalog.data.gov/dataset?q=%22national+bridge+inventory%22&sort=score+desc%2C+name+asc&ext_location=&ext_bbox=&ext_prev_extent=-139.21874999999997%2C-3.162455530237848%2C-61.87499999999999%2C55.57834467218206)
• Statistics of Income (http://catalog.data.gov/dataset?q=%22statistics+of+income%22&sort=score+desc%2C+name+asc&ext_location=&ext_bbox=&ext_prev_extent=-139.21874999999997%2C-8.05922962720018%2C-61.87499999999999%2C52.696361078274485)

I would highly recommend coordinating with the Federal statistical community on how collections might support them. Groups such as the Statistical Community of Practice and Engagement (SCOPE) will have helpful suggestions on how best to implement.

[philipashlock]

@dsmorgan77 the problems you described seemed to be well served by the model of having a child parent relationship where both the child and the parent would essentially be a fully qualified entry in a data.json. Does that answer your question @cew821?

So no, the children wouldn't be listed under the distribution. Instead the parent wouldn't list any distributions, but it might have a flag indicating that it's a collection, perhaps "collection":true or something and then each child would point to the parent with something like "collectionID":"uniqueid-12345"

Dan raises a good point about being able to be more precise about the temporal bounds for each individual file if something is released in fragments by region, but I think it would still be acceptable to package all those together as distribution under one dataset. This would still allow people to download individual files. You could argue that any dataset could be sliced into smaller pieces and create metadata for it, but some things are already logically packaged as a dataset, so I don't know that we really always need to create extra metadata for subsets like that.

[cew821]

Yes that makes sense. I think your proposal for additional optional data field indicating the parent makes sense (at least on the child record). I would think you would want to use identifier as the foreign key for the collectionID field (or maybe parentID would be a better name?). I'm not sure you would need to add a field to the "parent" record, but I suppose having some indicator that there are children records to go look for could be useful.

Adopting this approach would also open the possibility for nested parent/child relationships, which I think should be fine (i.e. a record could both be a child of a parent record, and itself be the parent of children records).

[lilybradley]

This is great. I like how the SKOS core guide deals with the "collection" issue: http://www.w3.org/TR/2005/WD-swbp-skos-core-guide-20051102/ - using broader/narrower/associated relationships and meaningful collections of concepts.

Works well with the standards that POD schema already employ.

In case you haven't seen it, HHS is using these standards for its catalog data.jsonld implementation:
"@context": {
"rdfs": "http://www.w3.org/2000/01/rdf-schema#",
"dcterms": "http://purl.org/dc/terms/",
"dcat": "http://www.w3.org/ns/dcat#",
"foaf": "http://xmlns.com/foaf/0.1/",
"pod": "http://project-open-data.github.io/schema/2013-09-20_1.0#"
},

[raking08]

The parent child issue also presents a different problem for my agency. In our case the parent dataset is often not releasable (PII or other reasons) so only the "derivative" or child data set is released. We are still obligated to have the parent/internal only dataset in the directory. The child is often not even stored on the same system (we have hundreds of custom systems for collecting and storing datasets). We are planning on using the unique identifier of the parent as an attribute of the derivative (open) dataset, but are only now considering whether that's looks like a concatenation of the parent U.I + a serialization number or if its two UIs(either way displaying as a collection on data,gov seems straightforward) As our internal catalogue will be in the form of a database that will transmit a file in the correct format, I am wondering how best to construct it in order to minimize the record maintenance. If its a full record, then there will be much redundancy of input which is a source of errors and loss of linkage to say nothing of data steward resistance to double inputs. As we consider the parent child issues is this a concern of anyone? This is not a big deal if 100 datasets, but we have estimated 8-10,000. Has anyone sorted this out?

[lilybradley]

Ranking08, could we solve that kind of problem with entity relationship diagrams (that we could somehow translate into a flat file)? ERDs are a lot like linked data. Similar solution for different problems.

Also NIST is building the NIEM. Your agency might be a good candidate for them as they move into beta.

[JoshData]

+1 for collections.

In HHS's DMS, each dataset is optionally given a Group Name, which is a string. Any datasets with the same group name are organized together in search results. We can map group names to collections in our data.json file, but we don't have any additional metadata for the group itself so we'd have to make up metadata for the parent dataset.

I think it might be simpler to put collection: ['childID', 'childID'] on the parent, and then there's no need for is_collection: true on the parent.

[raking08]

Thanks Josh.. one point in our case is the parent (for instance say it has PII) in most cases, will not be exposed on data.gov , or any other outward facing site, only the child... so the parent record will be in the internal "all dataset" catalogue and not in the public JSON file. Still, the children should all sort together .. so in this case the relationship should be maintained at the child record and not on the parent record... thoughts?

[JoshData]

@raking08: The parent could be a placeholder, I guess? It would be odd to have an identifier to something but the something isn't listed in the file. (Even if it has PII, it could still be listed but with accessLevel=restricted or nonpublic I think?)

[raking08]

interesting thought.. but there is significant resistance to having any exposure of the parent ( consider the rest of the metadata that would be exposed) so I do not think listing the parent would be allowed, but I will pose that to security. But as long as the children would be exposed with their different tags for spatial and temporal and any other specifics then it should be OK.. the placeholder could be interesting if there were say 100 children, but you would still need to be able to drill into the placeholder to get to the specific dataset you wanted. How do you see that working?

[cew821]

@JoshData I think the children need to have their "parentID" (i.e. the foreign key) in their record, since it's a one -> many relationship between parent and children, no? I suppose you could also include a list of all the children in a collection: [child1_id, child2_id] array in the parent record, as a convenience, but I would definitely expect the children records had a parent: parent_id field.

[cew821]

@raking08 I agree with the placeholder idea - even if all the record contained was a title, unique_id that matched with the children's parent_id and a accessLevel: non-public field, that would probably be enough for the public to understand that the children datasets were a public subset of this larger, non-public dataset.

[raking08]

Hi CEW,
Those are my thoughts too, in fact I am modeling this in separate tables so that the child can inherit characteristics of the parent without needing double data maintenance, but still output the flat file to the public exposure site. Due to additional business requirements for data lifecycle management, we may have an additional layer below the child as well in the internal repository, but it wont be public site relevant. I would be very curious to know if anyone has an internal system that combines managing the core metadata for all data sets ( internal and external) together with data lifecycle management functions.

[dsmorgan77]

@raking08 I don't see the problem with your particular instance of mentioning a dataset that has PII in your public data listing. I know it's optional, but the public already has notice that there is a system that collects data containing PII. How? Because your agency is publishing Privacy Impact Assessments and System of Records Notices telling them just that. In short, the public already knows.

And, the notion that the minimum required metadata is somehow "too much" metadata is kind of ludicrous. The title, description, and contact point (for a privacy-sensitive dataset, that'd just be the redress point of contact), and agency identifiers are all innocuous and, again, already made public when you release a PIA and a SORN.