x/mod/sumdb: Server.ServeHTTP serves data tiles in different format than sum.golang.org

[mjl-]

Go version

n/a (go1.23.1 on linux/amd64)

Output of go env in your module/workspace:

n/a

What did you do?

I used https://pkg.go.dev/golang.org/x/mod@v0.21.0/sumdb#Server.ServeHTTP to write a sumdb server for local testing.

I already had earlier code that parses data tiles (level -1 tiles) into records, and then verifies the hash of each record. It's part of gopherwatch, with the parsing at https://github.com/mjl-/gopherwatch/blob/3d50ff988dade2e571f3983c4194e451515a39d1/sums.go#L288. The parser is a bit sloppy, but it can verify the data tiles served by sum.golang.org.

Configuring gopherwatch to verify the data tiles of a sumdb server written with sumdb#Server.ServeHTTP was giving me unexpected verification errors (hashes of record data different than expected based on the tiles containing hashes).

It turns out that data tiles served by sum.golang.org have a different format than the data tiles served by golang.org/x/mod/sumdb.Server. In sumdb.Server, each record is prefixed with its record ID and a newline. On sum.golang.org they are not. The hashes of the records stored in the level 0 tiles are calculated over the record data without a record ID, that explains the errors I was seeing.

For example, a (random) data tile from sum.golang.org: https://sum.golang.org/tile/8/data/x115/673

It looks like this:

github.com/hewo233/cxsj_homework1 v0.0.0-20240826052307-bcb3e2e71e36 h1:QRoUMkmrcJluJBuUNVVVDUfaB1YyFlgbbZwDwrjPrss=
github.com/hewo233/cxsj_homework1 v0.0.0-20240826052307-bcb3e2e71e36/go.mod h1:6C5wTFANZQ6jZIu6UdOkia0qn8b4fQ+HFLF4CijMthE=

github.com/arkadiyt/bounty-targets-data v0.0.0-20240908180608-4f76f96365cf h1:vPLwkW3FLd+R2haNqSfa3CMizSYn2nSEaUyGg12aEeI=
github.com/arkadiyt/bounty-targets-data v0.0.0-20240908180608-4f76f96365cf/go.mod h1:0QqDOBpcXYlM/8MU2nsGjwvUOgJkKrVA/RA5hxf078s=

github.com/jaganathanb/dapps-api v0.0.0-20240617075156-dbbed11228b4 h1:IHcmr6+bukh2ZCyKbH2VkVfUsJGiwTYZKRbP7WCb7n8=
github.com/jaganathanb/dapps-api v0.0.0-20240617075156-dbbed11228b4/go.mod h1:ZBo/O1yJ08TwyEtzMQIQDL8dQz4vyLyW4m1Fss3WDDA=

...

However, a data tile served by sumdb.Server would look like this (note the extra number before the record data: the record id):

1234
github.com/hewo233/cxsj_homework1 v0.0.0-20240826052307-bcb3e2e71e36 h1:QRoUMkmrcJluJBuUNVVVDUfaB1YyFlgbbZwDwrjPrss=
github.com/hewo233/cxsj_homework1 v0.0.0-20240826052307-bcb3e2e71e36/go.mod h1:6C5wTFANZQ6jZIu6UdOkia0qn8b4fQ+HFLF4CijMthE=

1235
github.com/arkadiyt/bounty-targets-data v0.0.0-20240908180608-4f76f96365cf h1:vPLwkW3FLd+R2haNqSfa3CMizSYn2nSEaUyGg12aEeI=
github.com/arkadiyt/bounty-targets-data v0.0.0-20240908180608-4f76f96365cf/go.mod h1:0QqDOBpcXYlM/8MU2nsGjwvUOgJkKrVA/RA5hxf078s=

1236
github.com/jaganathanb/dapps-api v0.0.0-20240617075156-dbbed11228b4 h1:IHcmr6+bukh2ZCyKbH2VkVfUsJGiwTYZKRbP7WCb7n8=
github.com/jaganathanb/dapps-api v0.0.0-20240617075156-dbbed11228b4/go.mod h1:ZBo/O1yJ08TwyEtzMQIQDL8dQz4vyLyW4m1Fss3WDDA=

...

The data tiles are composed at https://github.com/golang/mod/blob/46a3137daeac7bd5e64dc5971191e4a7207e6d89/sumdb/server.go#L148,
using tlog.FormatRecord from https://github.com/golang/mod/blob/46a3137daeac7bd5e64dc5971191e4a7207e6d89/sumdb/tlog/note.go#L75.
The comment on that function also says that format is used by a data tile. That is true for sumdb.Server, but not for sum.golang.org.

The record format (with signature) is used in the responses to individual module lookups, e.g. https://sum.golang.org/lookup/github.com/mjl-/ding@v0.5.2.

The Go tooling may not be parsing the data tiles, working only on lookups and the tiles containing hashes. But I think it would be good to have sumdb.Server serve the same format as sum.golang.org for data tiles, so clients that can process data tiles from sum.golang.org can also parse data tiles served by a sumdb.Server.

There could be concerns about compatibility: existing code may now depend on the format currently served by sumdb.Server. But parsing data tiles doesn't seem very common.

If the format of sumdb.Server is deemed better and should stay, it would be good to document this difference. Clients parsing the data tiles could then recognize both formats.

What did you see happen?

sumdb.Server.ServeHTTP serving data tiles in a format different from what sum.golang.org is serving.

What did you expect to see?

sumdb.Server.ServeHTTP serving data tiles in the same format as sum.golang.org.

[timothy-king]

CC @matloob @samthanawalla as owners of x/mod.

[adonovan]

cc @rsc, as author of this package.

[FiloSottile]

Thank you for the thorough report! Indeed, it looks like tlog.FormatRecord reflects the /lookup format but not the sumdb data tiles. I think the x/mod/sumdb package should mirror the sum.golang.org implementation, so I'm in favor of editing the comment of tlog.FormatRecord not to mention data tiles, and changing the served format of Server.ServeHTTP.

I doubt anyone is depending on the current format since no one noticed the discrepancy before.

[gopherbot]

Change https://go.dev/cl/618135 mentions this issue: sumdb: make data tiles by Server compatible with sum.golang.org