Single-file /data catalog not good--optional alternative suggested

[jeffdlb]

Current guidance is that each agency's "/data" inventory must be a single list in a file containing multiple lines of Javascript Object Notation (JSON) summary metadata per dataset, even if our agency has tens of thousands of datasets distributed across multiple facilities and servers. I believe the single list will pose problems of inventory creation, maintenance, and usability. I enumerate my concerns below, but first I propose a specific solution.

PROPOSAL:

I recommend the single-list approach be made optional. Specifically, I suggest that the top-level JSON file be permitted to include either a list of datasets or a list of child nodes. Each node would at minimum have 'title' and 'accessURL' elements from your JSON schema (http://project-open-data.github.io/schema/), an agreed-upon value of 'format' such as "inventory_node" to indicate the destination is not a data file, and optionally some useful elements (e.g., person, mbox, modified, accessLevel, etc) describing that node. Each node could likewise include either a list of datasets or a list of children.

CONCERNS REGARDING THE SINGLE-LIST APPROACH:

(1) We should not build these inventories only to support data.gov. We want to leverage this for other efforts internal to our agencies, for PARR, and to support other external portals such as the Global Earth Observing System of Systems (GEOSS) or the Global Change Master Directory (GCMD). A distributed organization will be more useful for them (even if data.gov itself could handle a single long unsorted list.)

(2) The inventory will need to be compiled from many different sources, including multiple web-accessible folders (WAFs) of geospatial metadata, existing catalog servers, or other databases or lists. Each type of input will need some code to produce the required subset of the list, and then some software will need to merge everything into a giant list. Failure at any step in the process may cause the inventory to be out of date, incomplete, or broken entirely. A distributed organization will more easily allow most of the inventory to be correct or complete even if one sub-part is not, and will require much less code for fault-handling.

(3) Some of our data changes very frequently, on timescales of minutes or hours, while other data are only modified yearly or less frequently. A distributed organization will more easily allow partial updates and the addition (or removal) of new collections of data without having to regenerate the entire list.

(4) The inventory is supposed to include both our scientific observations and "business" data, and both public and non-public data. That alone suggests a top-level division into (for example) /data/science, /data/business, and /data/internal. The latter may need to be on a separate machine with different access control.

(5) It would be easier to create usable parallel versions of the inventory in formats other than JSON (e.g., HTML with schema.org tags) if the organization were distributed.

(6) I understand that the data.gov harvester has successfully parsed very long JSON files. However, recursive traversing of a web-based directory tree-like structure would be trivial to implement by data.gov and would be more scalable and solve many problems for the agencies and the users. data.gov's own harvesting could even be helped if the last-modified date on each node is checked to determine whether you can skip it.

[waldoj]

@jeffdlb, is it fair to say that the Sitemap index standard is a good model here?

[mhogeweg]

@jeffdlb @waldoj I'd say so and had done so before in issue #27 where I suggested pagination like is seen in OpenSearch providers or the breaking up in smaller files as done in sitemap. I think Jeff's point about supporting needs not just of Data.gov but also think about how agencies support other initiatives also deserves consideration here.

[benbalter]

I'd argue that the schema was written with developers in mind, not the ease of agency adoption (or data.gov). When the two are in conflict, we should err on the side of those we want to encourage to use the data, not those whose job it is to publish or organize the data. Going into multiple formats may make things easier for agencies, but it does so at the average developer's expense.

This is a great case for practicality over purity. A single .data.json file that's too large to easily manipulate would be a great problem to have. It would mean agencies are indexing data and exposing it to the public, but as far as I can tell, looking through the open issues here, that problem remains theoretical and limited to government. Given the ease of adopting options like #27, I'd argue for a wait and see approach. Let the users' needs drive the product, not the publishers.

Practically, if we allow sub-data files two implications:

1. A lot more complex for developers (or data.gov or whatever) to crawl. The HTTP call is going to be the most expensive part of the transaction. Allowing sub-files doubles that transaction cost, not to mention the complexity of the crawler.
2. If I were a government agency, I‘d take advantage of it. I’d make a single data.json file in the root, and then have each bureau/office make their own data.json file, so there'd be no change whatsoever from the status quo where data is siloed.

Forcing agencies to make a single .data.json file is an exercise that helps the agency centralize their data index, to absorb the complexity of their own system on behalf of citizens, and begins the process of becoming more customer-centric when it comes to data.

[waldoj]

Each type of input will need some code to produce the required subset of the list, and then some software will need to merge everything into a giant list. Failure at any step in the process may cause the inventory to be out of date, incomplete, or broken entirely. A distributed organization will more easily allow most of the inventory to be correct or complete even if one sub-part is not, and will require much less code for fault-handling.

But doesn't the process that you're describing still necessitate that "some software will need to merge everything into a giant list"? It seems that complexity is being pushed out to the clients, rather than being resolved within the federal agency.

[ddnebert]

A subtlety in this proposal, and one I opened as a suggestion in github two months ago, is that we allow the data.json file to contain entries for existing standards-based catalogs or APIs. Each such collection - and we manage many in the geospatial domain - could be simply marked up, one entry per collection, within the agency's json file with the breadcrumbs to perform the query and or indexing. CKAN already has built-in harvesting capability on these established protocols, as well as json, so the integration challenge would be minor. It would produce the same, if not better results as a serialized subset of metadata in feeds, since it all ends up in the searchable index. The benefits of this approach are many:

1. Geospatial metadata are robust and support not only discovery but fitness-for-use information that end-users need to know before access. The json file does not contain sufficient metadata for end-user advice.
2. The hybrid solution supports standards and gains access to well over 90% of existing government data sets in the catalog.data.gov index. Proposing an alternative, non-standard solution does not provide new content or augment counts of data.
3. Data conversion burdens on the agencies are negligible. This is an elegant, least-effort solution as support for these protocols and formats is built into the CKAN index software already.
4. The hybrid solution supports traversing and indexing homogeneous data series (i.e. imagery and similar inventory catalogs) in a two-phase search, a feature not present in the json solution.
5. The catalog solution supports detection of resource changes (add, update) not supported in the json feed. Full traversal/re-indexeing of complete agency json files is currently required, whereas change detection is already supported in the protocols and harvest.

I propose the following text changes to the Implementation document, implementation-guide.md, along with a modification of the harvest routine to recognize the catalog resource type within the feed:

A) Minimum Required for Compliance

Produce a single catalog or list of data managed in a single table, workspace, or other relevant location. Describe each dataset or existing metadata catalog according to the common core metadata.

This listing can be maintained in a Data Management System (DMS) such as the open-source CKAN platform; a single spreadsheet, with each metadata field as its own column; or a DMS of your choosing. A description of each agency metadata catalog, such as CKAN, can be placed in the agency json file as a single entry. This entry will describe the resource type of "catalog" and the access URL to be used in harvest by data.gov.

Metadata for geographic or geospatial information is often collected using the FGDC Content Standard for Digital Geospatial Metadata or ISO 19115/19139 and represented as XML, providing content that maps to common core metadata. These collections are exposed using the Open Geospatial Consortium Catalog Service for the Web interface (CSW 2.0.2) or as a read-enabled HTTP directory known as a Web Accessible Folder (WAF). In lieu of posting individual entries for each geospatial dataset in the json file, a single json entry should be prepared for each geospatial metadata collection (WAF) or service (CSW) as a "Harvest Source" enabling harvest of the collections by catalog.data.gov. Individual geospatial metadata entries for datasets, applications, or services should not be duplicated in the agency json feed.

[mhogeweg]

thanks @ddnebert for pointing out that agencies have FGDC/ISO metadata. I had posted a mapping of those specs to DCAT (as the mapping is not trivial with absence of 1:N elements in DCAT, different dates notations, and different interpretations of fields to name a few) and submitted that as pull request #74. would like to see your thoughts on that mapping.

A second point is the focus on getting things to work for CKAN. I understand that Data.gov uses CKAN, but would it not be better if a solution is designed that works across the government regardless of technology? That is what the geospatial domain has been working on for many years and what was done to promote an open ecosystem of suppliers and consumers of data. To me that also relates to @jeffdlb's point regarding programs like GEOSS (which you play a key role in) and GCMD, not to mention Eye on Earth, UNEP Live and various other global initiatives focused on open data sharing.

[jeffdlb]

@waldoj - I don't think breaking up the single list into a linked set of lists pushes complexity to the users. The end goal of the inventory is not just to have a list, it is to populate data.gov or other portals or commercial search engines. In all cases, entries in each list (whether there is one list per agency or many) will be going into a database of some type. That database will be updated one entry at a time by reading through the lists.

[jeffdlb]

@benbalter - If every bureau's sub-list is linked from the master list, then I believe the data would be less "siloed" than currently. At present any bureau-level inventories are not standardized, whereas this effort would standardize and link them.

[cew821]

@ddnebert I like this suggestion. In fact, some geospatial energy data has already been posted to data.gov using this approach (CKAN harvesting of a CSW endpoint). See below for details of this example:

The Department of Energy is helping create a datastore for geothermal data called the National Geothermal Data Store. One of the nodes of this system is the State Geothermal Database. This data store conforms to the ISO 19139 geospatial metadata standard discussed above, and uses the Catalog Service for the Web (CSW) standard to provide interoperability. Here's an example CSW endpoint to the same data store linked to above.

Because the CKAN harvester knows how to inter-operate with the CSW, all of the datasets (including the geospatial data) were able to be added to data.gov just by pointing the harvester at the CSW endpoint. End result? A search for 'borehole temperatures' on data.gov yields 59 datasets, with intact metadata, available in Esri REST API, WMS, WFS, and ZIP file formats.

For agencies that have existing data portals that can be easily harvested by CKAN, pointing the harvester to the data store itself could make a lot of sense.

I should note that not ALL of the Department's data will be easily harvested in this way. For legacy or custom data portals that did not use standards, or are otherwise not up to date, creating the data.json file is a good forcing function. But for systems that have this capability, why not use it?

[waldoj]

CKAN's CSW syndication functionality is great, but this initiative is not about expanding CKAN support within the federal government. The goal is to "produce a single catalog or list of data managed in a single table, workspace, or other relevant location". Putting some data behind a different protocol runs counter to this goal (the data then ceases to be "in a single table"), and erects a significant hurdle to anybody who wants to syndicate that data (requiring parsing 2+ catalog formats).

Perhaps some tools could be built to solve this problem via both design patterns? That is, the need to syndicate data can be address via harvesting from CSW (for example), precisely as proposed here. Simultaneously, some tools can be built to extract that catalog data and convert it into the data.json format, alongside the existing tools to aid in this transition. I'd be happy to scrub in to help create such a converter!

[MarionRoyal]

Very good discussion. If I might add my couple of pennies...

History

The original DataGov metadata template created back in 2009 was done rather
quickly and at that time DCAT really hadn't taken hold and
schema.orgwasn't even an idea. The success of the original template
was demonstrated
by the fact that we only had one major revision (congratulations to the
multi-agency team who developed it.) Based on Dublin Core, it clearly had
the basic metadata (and some fields that were never used) needed to express
the basic concept of a stored chunk of data. If we could have found a
simple standard to use instead of building our own, we would have done so.
To be fair, though, we had some administrative requirements such as "Does
this Dataset meet your agencies compliance to the Data Quality Act?" which
would not likely be found in any other standard.
It was well understood that the DataGov metadata template was a temporary
solution and not a long term "Standard".

DCAT evolved as a means of connecting catalogs together and in doing so,
harmonized most of the metadata catalog terms (and incorporated other
namespaces like DC and FoaF). Not different enough from the original
DataGov template to make a change but notable as we considered mapping to
other schema's.

When we started tying DataGov with Geospatial One-Stop (GOS), we used an
API on the Geo Catalog to map and display the metadata at Data.gov using
the DataGov template. This was fine as long as there was a rich metadata
catalog accessible for geospatial mapping and storage of map services (I
have now reached my extent of knowledge of geo-speak).
The Geospatial metadata was well established before DataGov albeit with an
FGDC evolution to ISO 19115 (I have now reached my extent of knowledge of
FGDC vs ISO). Simple mapping from the geospatial metadata to DataGov
template was never a problem. However, the DataGov template was a
_SUBSET_of all of the fields found in either FGDC or ISO. The fields
used in the
geo community were way too specialized to even be considered for non-geo
records.

We kept this rich catalog of metadata along with it's harvesting
capability as we brought together DataGov and GOS and duplicated their
infrastructure at geo.data.gov. We used the functionality of the
geo.data.gov catalog as a requirement as we deployed and contributed to the
CKAN software. The end result is that catalog.data.gov supports harvesting
of FGDC and ISO and the basic catalog requirements of GeoPlatform.gov
(thank you Doug et al).

There are experts already in this discussion who knows the details
exponentially to me, so I won't dally around. It is important for those
who do not know the history to poke around a bit to understand the
complexity of the geo community.

So

We are asking the geospatial community to abandon their long-fought process
of establishing an international metadata standard through ISO (this
process is a career, not a project) and adopting a NEW CORE schema in a way
that is better than the original DataGov metadata template but does not
contain the rich catalog information needed for their community.

They just can't do that. Of course they can develop tools to spit out
the JSON metadata (subset) from their records. We can do that from
Data.gov. But what value is it? It doesn't contain the metadata that any
geo-scientist would need. They would still need to maintain the FGDC/ISO
records. It would put us in a situation where we are treading down the
path to separate catalogs which is the opposite direction that we need to
be headed in.

Today

Today we are preparing to harvest agency metadata using the new CORE schema
which has been validated against the original DataGov metadata template and
is obviously an improvement. The new CORE schema is a SUBSET of a
mapping to FGDC/ISO. At DataGov, we will continue to harvest geospatial
metadata using FGDC and/or ISO19115 (even if the subset of metadata is in
the JSON file), because that's the requirement for GeoPlatform.gov and
other geo-scientists.

Reality (from my perspective)

The new CORE metadata schema is a temporary solution, not a permanent one.
It will not become an ISO standard. It will not supersede Dublin Core.
It will not even supersede DCAT. It is not even based in a well known
namespace.

However, it may be what we need just now; a simple way for agencies to make
their data available in a way that the public understands it (and by
public, I include developers.) It also flips around the manner is which
these data are published in a way that applications (other than CKAN and
Data.gov) can parse the information and make use of it.
Shall we expand this temporary solution in a way that it meets the
requirements of all? Heck no!

Instead, we should seek a longer term solution (maybe not permanent but
certainly more scalable.) Is the long term solution ISO19115? Sorry no.

I think that the long term solution that we should be working toward is *
SCHEMA.ORG*. I won't go into the reasons. That's a whole different
discussion and if I don't make everybody mad, I would love to be part of
that work.

So What (in my opinion)

The proposal that this simple (long) JSON file may contain pointers to more
complex catalogs seems like a reasonable approach to me. Developers should
be sophisticated enough to recognized a field that states this pointer will
require a different parsing mechanism and either ignore or pursue. In the
mean time, let's continue our goal for a long term solution and make that
solution the best it can be.

On Mon, Aug 5, 2013 at 10:42 PM, Waldo Jaquith notifications@github.comwrote:

CKAN's CSW syndication functionality is great, but this initiative is
not about expanding CKAN support within the federal government. The goal is
to "produce a single catalog or list of data managed in a single table,
workspace, or other relevant location"http://project-open-data.github.io/implementation-guide/.
Putting some data behind a different protocol runs counter to this goal
(the data then ceases to be "in a single table"), and erects a significant
hurdle to anybody who wants to syndicate that data (requiring parsing 2+
catalog formats).

Perhaps some tools could be built to solve this problem via both design
patterns? That is, the need to syndicate data can be address via harvesting
from CSW (for example), precisely as proposed here. Simultaneously, some
tools can be built to extract that catalog data and convert it into the
data.json format, alongside the existing tools to aid in this transitionhttp://project-open-data.github.io/#4_tools.
I'd be happy to scrub in to help create such a converter!

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/105#issuecomment-22155058
.

Marion A. Royal PMP
Program Director, DataGov
GSA Office of Citizen Services and Innovative Technologies
202.302.4634

[ddnebert]

Regarding the statement: "CKAN's CSW syndication functionality is great, but this initiative is not about expanding CKAN support within the federal government. The goal is to "produce a single catalog or list of data managed in a single table, workspace, or other relevant location". " it is my understanding that the intention of the .json feeds was to feed the single government data search engine (powered by CKAN) to create a comprehensive view of governmental metadata. The creation of a feed (or catalog service) is only a means to an end since the metadata cache at CKAN provides the necessary interface to access all governmental metadata, or selected metadata via query. BTW, CKAN has the ability to expose its results in json and RDFa, albeit a DCAT flavor, as the result of a query. So, if it is json format that you need, it is also accessible there in addition to several other APIs.

By allowing multiple (already supported) protocols and formats, we have now created a seamless virtual catalog of government metadata. We require the option to reference the existence of metadata collections as WAF or CSW individual entries within the agency json feed. This is an operational solution that fulfills the requirements of both the geospatial and 'raw' data communities - allowing agency and federated views, exposing actionable APIs, enabling counts and tracking, and most importantly enabling access to the data, services, and applications of our federal, state, local, tribal, and academic partners.

[waldoj]

it is my understanding that the intention of the .json feeds was to feed the single government data search engine (powered by CKAN) to create a comprehensive view of governmental metadata.

The stated purpose of providing the specified data.json file, as described in Slash Data Catalog Requirements, is "to make [the catalog] more easily discoverable to private-sector developers and entrepreneurs." Data.gov is just one client.

[ddnebert]

The proposed hybrid solution that feeds a common search and retrieval API (that can return json and other formats) supports that requirement even better. With /data in every agency, developers would need to locate such directories and then visit each one individually. With a common search interface built on the json syndication, developers will have an easier time interacting with the metadata through a single entrypoint.

[waldoj]

I follow, but M-13-13 says:

Any datasets in the agency’s enterprise data inventory that can be made publicly available must be listed at www.[agency].gov/data in a human- and machine-readable format

That's a per-agency mandate. There are lots of details about implementation that can be altered, but fundamentally M-13-13 requires a complete inventory, on the agency website, within /data, as human- and machine-readable data.