Step 5: data dictionary and Croissant metadata
- data/dictionary.yaml: every published file, and every column of runs_clean.csv with its type, meaning and allowed values - collection/publish_metadata.py writes data/croissant.json (Croissant 1.0) from it, with a sha256 per file; `npm run metadata`, run by the daily update - tests/core/test_dictionary.py: every column and value described, types hold, croissant.json up to date and checksums match - attempt is now written as a whole number (was 1.0, 2.0); no statistic changes - the site ships both files and the Methods text links them Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
11b4a0398e
commit
63b5d40e54
@@ -8,3 +8,11 @@
|
||||
| Reddit megathreads (r/fourthwing, Dragonkind bonding Parts 3 and 4) | qualitative; routes cited in the valley map | manual |
|
||||
| DragonKind bot exports | `bot/runs-YYYY-MM-DD.jsonl` | by pull request, see `docs/BOT_DATA_PROTOCOL.md` |
|
||||
| Discord | not yet | planned, with admin consent and anonymised usernames |
|
||||
|
||||
## What each column means
|
||||
|
||||
`data/dictionary.yaml` describes every published file and every column of `data/derived/runs_clean.csv`: its type,
|
||||
what it means and every value it can take. `data/croissant.json` says the same in the
|
||||
[Croissant](https://docs.mlcommons.org/croissant/) metadata format (with a sha256 checksum for every file), so
|
||||
dataset tools can load and cite the data. After changing the data or the dictionary, run `npm run metadata`; the
|
||||
tests fail if a column or value is not described or the metadata is out of date.
|
||||
|
||||
Reference in New Issue
Block a user