CI / test (pull_request) Successful in 38s
The daily check keeps the community route PDF only in its private raw store (never published here). When that copy differs from "Threshing Options.pdf" in the bot's private repo, `python -m collection send-pdf` opens (or updates) a pull request there with the new PDF and the lines its text gained and lost, so Ken can update the hand-written routes in src/pdfRoutes.js (the bot does not read the PDF itself). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UCzTd31GphpMXDhQqWrCMQ
115 lines
6.0 KiB
YAML
115 lines
6.0 KiB
YAML
# Every day: check every source (collection/pipeline.yaml), and when something changed, rebuild the runs, the
|
|
# graph, the route model and the statistics, run the tests and open a pull request that merges itself once CI
|
|
# passes. A source that cannot be reached, or new answers the coding rules do not know, open an issue instead.
|
|
# Gitea Actions; GitHub's copy (with GitHub Pages) is .github/workflows/update-sources.yml.
|
|
#
|
|
# Secrets (repository Settings -> Actions -> Secrets; docs/REPO_SETTINGS.md):
|
|
# ATLAS_BOT_TOKEN a Gitea access token (repository: read and write; issue: read and write). Needed to push the
|
|
# update branch and open the pull request; the built-in token cannot start the CI it needs.
|
|
# RAW_ARCHIVE_KEY optional: encrypts the private raw store kept in the Actions cache between runs
|
|
# REDDIT_CLIENT_ID, REDDIT_CLIENT_SECRET optional: read the megathreads through the Reddit API
|
|
# DK_ACCOUNT_SALT the salt that turns bot account names into anonymous keys (keep it the same forever). With it,
|
|
# the bot's private repo (DragonKind/DragonKind) is downloaded and imported; without it, skipped.
|
|
# DRAGONKIND_TOKEN optional: a read-only token for the bot's repo (ATLAS_BOT_TOKEN is used otherwise)
|
|
name: Check sources for new data
|
|
on:
|
|
schedule:
|
|
- cron: "23 10 * * *" # daily
|
|
workflow_dispatch:
|
|
jobs:
|
|
check:
|
|
runs-on: ubuntu-latest # a runner label on the server (act_runner)
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
token: ${{ secrets.ATLAS_BOT_TOKEN }}
|
|
- uses: actions/setup-node@v4
|
|
with:
|
|
node-version: 20
|
|
- uses: actions/setup-python@v5
|
|
with:
|
|
python-version: "3.12"
|
|
- run: pip install -r requirements.txt
|
|
- name: Restore the private raw archive
|
|
uses: actions/cache/restore@v4
|
|
with:
|
|
path: ${{ runner.temp }}/raw-archive
|
|
key: raw-archive-none
|
|
restore-keys: raw-archive-
|
|
- name: Decrypt it
|
|
env:
|
|
RAW_ARCHIVE_KEY: ${{ secrets.RAW_ARCHIVE_KEY }}
|
|
run: |
|
|
f="$RUNNER_TEMP/raw-archive/raw.tgz.enc"
|
|
if [ -n "$RAW_ARCHIVE_KEY" ] && [ -f "$f" ]; then
|
|
openssl enc -d -aes-256-cbc -pbkdf2 -pass env:RAW_ARCHIVE_KEY -in "$f" | tar -xz -C data
|
|
fi
|
|
- name: Download the bot's data (DragonKind/DragonKind, private)
|
|
env:
|
|
DK_TOKEN: ${{ secrets.DRAGONKIND_TOKEN || secrets.ATLAS_BOT_TOKEN }}
|
|
DK_ACCOUNT_SALT: ${{ secrets.DK_ACCOUNT_SALT }}
|
|
run: |
|
|
if [ -n "$DK_TOKEN" ] && [ -n "$DK_ACCOUNT_SALT" ]; then
|
|
GIT_CONFIG_COUNT=1 GIT_CONFIG_KEY_0="http.https://git.kenschiano.net/.extraheader" \
|
|
GIT_CONFIG_VALUE_0="AUTHORIZATION: token $DK_TOKEN" \
|
|
git clone --quiet --depth 1 https://git.kenschiano.net/DragonKind/DragonKind.git "$RUNNER_TEMP/dragonkind"
|
|
echo "DRAGONKIND_BOT_DIR=$RUNNER_TEMP/dragonkind" >> "$GITHUB_ENV"
|
|
else
|
|
echo "DK_ACCOUNT_SALT is not set: the bot source is skipped"
|
|
fi
|
|
- id: check
|
|
env:
|
|
ATLAS_BOT_TOKEN: ${{ secrets.ATLAS_BOT_TOKEN }}
|
|
DK_ACCOUNT_SALT: ${{ secrets.DK_ACCOUNT_SALT }}
|
|
REDDIT_CLIENT_ID: ${{ secrets.REDDIT_CLIENT_ID }}
|
|
REDDIT_CLIENT_SECRET: ${{ secrets.REDDIT_CLIENT_SECRET }}
|
|
run: python -m collection update --report "$RUNNER_TEMP/report.md"
|
|
- name: Rebuild the analysis and run tests
|
|
# not when the combined runs failed validation: the issue below reports the new answers instead
|
|
if: steps.check.outputs.changed == 'true' && !contains(steps.check.outputs.failed, 'build')
|
|
run: |
|
|
python -m collection rebuild
|
|
npm test
|
|
- name: Encrypt the raw store
|
|
id: archive
|
|
env:
|
|
RAW_ARCHIVE_KEY: ${{ secrets.RAW_ARCHIVE_KEY }}
|
|
run: |
|
|
if [ -n "$RAW_ARCHIVE_KEY" ] && [ -d data/raw ]; then
|
|
mkdir -p "$RUNNER_TEMP/raw-archive"
|
|
tar -czf - -C data raw | openssl enc -aes-256-cbc -pbkdf2 -salt -pass env:RAW_ARCHIVE_KEY -out "$RUNNER_TEMP/raw-archive/raw.tgz.enc"
|
|
echo "save=true" >> "$GITHUB_OUTPUT"
|
|
fi
|
|
- name: Save it (a new copy only when a raw file changed)
|
|
if: steps.archive.outputs.save == 'true'
|
|
uses: actions/cache/save@v4
|
|
with:
|
|
path: ${{ runner.temp }}/raw-archive
|
|
key: raw-archive-${{ hashFiles('data/manifest.json') }}
|
|
- name: Send a changed route PDF to the bot (its private repo; the PDF is never published here)
|
|
if: always() && steps.check.outcome == 'success'
|
|
continue-on-error: true
|
|
env:
|
|
ATLAS_BOT_TOKEN: ${{ secrets.ATLAS_BOT_TOKEN }}
|
|
run: python -m collection send-pdf
|
|
- name: Open a pull request with the new data (it merges itself once CI passes)
|
|
if: steps.check.outputs.changed == 'true' && !contains(steps.check.outputs.failed, 'build')
|
|
env:
|
|
ATLAS_BOT_TOKEN: ${{ secrets.ATLAS_BOT_TOKEN }}
|
|
run: |
|
|
# one branch per run: a second run on the same day (or a leftover branch) never blocks the push
|
|
branch="data/source-update-$(date -u +%Y%m%d)-run${{ github.run_number }}"
|
|
git config user.name "Atlas data bot"
|
|
git config user.email "atlas-bot@noreply.localhost"
|
|
git checkout -b "$branch"
|
|
git add -A data analysis/reports core/graph.js .github/ISSUE_TEMPLATE
|
|
git commit -m "data: update from community sources"
|
|
git push origin "$branch"
|
|
python -m collection forge open-pr --head "$branch" --title "New data from the community sources" --body-file "$RUNNER_TEMP/report.md" --automerge
|
|
- name: Report a broken source or new answers
|
|
if: always() && steps.check.outputs.failed != ''
|
|
env:
|
|
ATLAS_BOT_TOKEN: ${{ secrets.ATLAS_BOT_TOKEN }}
|
|
run: |
|
|
python -m collection forge open-issue --title "Source check failed: ${{ steps.check.outputs.failed }}" --body-file "$RUNNER_TEMP/report.md"
|