Walkthrough: From a Working Folder to a Pin#

This page is the week 6 demo, written down so that you can redo it after class. It uses the class report, the repository you work in for HW 4, and one of its upstreams, ftsfr/fed_yield_curve. Every command runs from a clone of the report, and every block of output is cut down from a real run. Steps A to D were run on November 10, against the pins the report has had since November 3. Step E’s output is from November 2, against the pin of that day, because it shows a day when the Fed had published new data since the pin.

The words are the ones ChartBook: Know Where Every Number Came From defines: pipeline, store, snapshot, lockfile, pin, reference, lineage. Read that page first; this one shows each word on the screen.

What you need. About an hour, and:

  • A clone of the report with its environment installed (uv sync in the clone). Activate it once, source .venv/bin/activate, so that the commands below are typed as written; or put uv run in front of each.

  • chartbook --version printing 0.5.1 or newer.

  • The snapshots chartbook.lock pins, in your store, each built by you (step E shows how, for one of them). chartbook pin --frozen passes when this is set up.

  • jq, for reading the lockfile (brew install jq, or your package manager).

A. The summer edition: a working folder#

What you are about to see: the Summer 2026 edition of the report, on the summer branch. It was built with ChartBook 0.1, which had no snapshots and no pins.

git switch summer
ls chartbook.lock
grep -n -A3 "data.load" src/chart_yield_curve.py
sed -n 140,152p README.md
ls: chartbook.lock: No such file or directory
34:    long = data.load(
35-        pipeline="ftsfr/fed_yield_curve",
36-        dataframe="ftsfr_treas_yield_curve_zero_coupon",
37-        format="pandas",
   chartbook catalog add ~/GitRepositories/ftsfr_repos/* -y

There is no lockfile. The exhibit reads by name, which is good, but the name resolved through a catalog of working folders on one machine: the README tells you to clone every upstream, run doit in each, and register the folders. Whatever those folders held on the day of the build is what the August PDF shows.

The point: which yield curve did the August report read? Git has the code. Nothing has the data. The folder it read from has been rebuilt since, so August’s bytes are gone, and no one can answer. Do not build summer with ChartBook 0.5 (it needs 0.1.1); switch back.

git switch main

B. The new main: a pin that names its bytes#

What you are about to see: the same exhibit, now reading through a pin. First, where ChartBook will look.

chartbook store info
Store default: /Users/you/.chartbook/store (writable)
  ftsfr/fed_yield_curve: 1 snapshot(s)
  ... (6 pipelines)
  Newest: 2026-11-04T06:04:56.031930Z

This is the default store, made when your first snapshot was. (Before chartbook catalog init, the same command prints the folder alone.) The report declares its upstreams in chartbook.toml:

sed -n '/^\[dependencies\]/,$p' chartbook.toml
[dependencies]
"ftsfr/fed_yield_curve" = { repository = "https://github.com/ftsfr/fed_yield_curve" }
"ftsfr/ken_french_data_library" = { repository = "https://github.com/ftsfr/ken_french_data_library" }
"ftsfr/he_kelly_manela" = { repository = "https://github.com/ftsfr/he_kelly_manela" }
"ftsfr/corp_bond_returns" = { repository = "https://github.com/ftsfr/corp_bond_returns" }
"ftsfr/options" = { repository = "https://github.com/ftsfr/options" }
"ftsfr/nyu_call_report" = { repository = "https://github.com/ftsfr/nyu_call_report" }

Each line names a pipeline by its ID, found by that ID in a store, with no folder of it on this machine. repository says where that pipeline’s code lives, so that a reader of this file knows what to clone and build; a pin then also keeps to snapshots made in that repository, should two ever claim the same ID. "ftsfr/fed_yield_curve" = {} pins the same snapshot, but write the address. (chartbook add https://github.com/ftsfr/fed_yield_curve is how such a line gets written: it adds the line, address included, and pins the newest snapshot in your stores.) What the report got for it is in chartbook.lock. Two records matter for one dependency: the dependency’s entry, and the record of each asset pinned from it.

jq '.records[0].dependencies[] | select(.id=="ftsfr/fed_yield_curve")' chartbook.lock
jq '.records[] | select(.id=="ftsfr/fed_yield_curve:ftsfr_treas_yield_curve_zero_coupon")' chartbook.lock
  "id": "ftsfr/fed_yield_curve",
  "via": "store",
  "location": { "kind": "store", "commit": "5bb7217c30f422bab5c9b87e3fc3410461ea4391",
                "repository": "https://github.com/ftsfr/fed_yield_curve", ... },
  "snapshot": "2026-11-04T05:46:01.793770Z",
  "pinned_hash": "sha256:29b80f24a77d83736a5bd0343abae385097bd47340c76b493530686d7e636280"

  "id": "ftsfr/fed_yield_curve:ftsfr_treas_yield_curve_zero_coupon",
  "location": { "kind": "file", "path": "./_data/ftsfr_treas_yield_curve_zero_coupon.parquet" },
  "content_hash": "sha256:4c04214626a01006a131ff11fa391453a52c6f1dc15d5daede2d7704642a2506",
  "size_bytes": 5376252

Four fields to know by name:

Field

Says

commit

Which code built it: the upstream’s commit, 5bb7217

snapshot

When the snapshot was made. This is the snapshot’s name, not when you pinned it; the lockfile records no time of its own

pinned_hash

One hash for the whole dependency: a hash over every asset pinned from it (step C)

content_hash

One hash per file: the SHA-256 of its bytes

Now the walk up from the lockfile:

chartbook lineage
finm-32800/finm_midterm_report (this folder)
├── ftsfr/fed_yield_curve: ftsfr/fed_yield_curve 5bb7217, made 2026-11-04 05:46:01 UTC, pins checked
│   ... (6 lines)
6 snapshot(s) upstream; no gaps.

Each line is a snapshot in a store, checked: it is there, it holds what was pinned from it at those hashes, and its own pins cover its own dependencies. Then read the data, and ask where it came from:

python -c 'from chartbook import data
df = data.load("fed_yield_curve:ftsfr_treas_yield_curve_zero_coupon", format="pandas")
print(df.shape, df.ds.max())
print(data.path("fed_yield_curve:ftsfr_treas_yield_curve_zero_coupon"))'
(382802, 3) 2026-10-30 00:00:00
<store>/pipelines/ftsfr+fed_yield_curve/2026-11-04T054601Z_5bb7217c30f4/_data/ftsfr_treas_yield_curve_zero_coupon.parquet

The path is in the store, under the snapshot’s name (its time and commit), and the report’s code never wrote it down: data.load resolved the reference through chartbook.lock. Both spellings of the reference work, fed_yield_curve:... and ftsfr/fed_yield_curve:...; the exhibits use the long one. doit watches that same file:

doit info exhibit:yield_curve
file_dep   :
 - ./src/settings.py
 - <store>/pipelines/ftsfr+fed_yield_curve/2026-11-04T054601Z_5bb7217c30f4/_data/ftsfr_treas_yield_curve_zero_coupon.parquet
 - ./src/chart_style.py
 - ./src/exhibits/yield_curve.py

dodo.py builds that list with data.path(...) for every reference in the exhibit’s UPSTREAM. Build it:

uv run doit
doit forget exhibit:yield_curve && doit
.  exhibit:yield_curve

A full build from clean takes about a minute; a second doit does nothing. After doit forget, only that exhibit reruns, and nothing after it: the PNG came out byte-identical, and doit compares inputs by hash. Open reports/class_report.pdf at the yield curve page: those are the numbers.

The point: the report names data, never a path. The path comes from the pin, doit watches the pinned file, and the exhibit reruns when, and only when, the pin moves. Data validation would catch bad data; the pin catches different data.

C. The hashes, by hand#

What you are about to see: every hash in the lockfile can be recomputed with shasum and jq. First the file:

shasum -a 256 "$(python -c 'from chartbook import data; print(data.path("fed_yield_curve:ftsfr_treas_yield_curve_zero_coupon"))')"
4c04214626a01006a131ff11fa391453a52c6f1dc15d5daede2d7704642a2506  <store>/.../ftsfr_treas_yield_curve_zero_coupon.parquet

That is the content_hash from step B. Then the dependency’s hash, which is a hash over a listing, one line per pinned asset, <hex>  <ID>, sorted by ID. Write the listing first, then hash it:

jq -r '.records[]
    | select(.pipeline == "ftsfr/fed_yield_curve" and .content_hash)
    | "\(.content_hash | ltrimstr("sha256:"))  \(.id)"' chartbook.lock \
    | LC_ALL=C sort -k2
a2eb227eec63003141a823dddac484f27a320749a2061140c5a42f7532abadb7  ftsfr/fed_yield_curve:chart/yield_curve_replication
4c04214626a01006a131ff11fa391453a52c6f1dc15d5daede2d7704642a2506  ftsfr/fed_yield_curve:ftsfr_treas_yield_curve_zero_coupon
103efb4b65d78515b9c4434fc6f75aca17e20298db6d5db7d7bb4d382959e53a  ftsfr/fed_yield_curve:notebook/gsw_crsp_replication_ipynb
cf4e6b4f98afa7705141ca6b8e13bfa56ef0514cda5b48c08e8a2e5004d70a45  ftsfr/fed_yield_curve:notebook/summary_fed_yield_curve_ipynb

Four assets: a chart, the dataframe, two notebooks. Pipe the same listing into shasum, and read the lockfile’s answer next to it:

jq -r '.records[]
    | select(.pipeline == "ftsfr/fed_yield_curve" and .content_hash)
    | "\(.content_hash | ltrimstr("sha256:"))  \(.id)"' chartbook.lock \
    | LC_ALL=C sort -k2 | shasum -a 256
jq -r '.records[0].dependencies[] | select(.id == "ftsfr/fed_yield_curve") | .pinned_hash' chartbook.lock
29b80f24a77d83736a5bd0343abae385097bd47340c76b493530686d7e636280  -
sha256:29b80f24a77d83736a5bd0343abae385097bd47340c76b493530686d7e636280

The same hash. (The recipe matches for all six dependencies in the report’s lockfile.) Change one byte of one file upstream and the file’s content_hash changes, so the listing changes, so the pinned_hash changes: one line of a diff says something moved.

Now git, for contrast. A commit is a hash over its tree and its parent; a tree is a hash over the names and hashes of what is inside it.

git cat-file -p HEAD
git cat-file -p 'HEAD^{tree}' | head -12
git rev-parse HEAD:chartbook.lock
printf 'blob %s\0' $(wc -c < chartbook.lock | tr -d ' ') | cat - chartbook.lock | shasum -a 1
tree df84ef48f8d89245a81da7b13581f98ae4c7b4b5
parent c914aea725d756022dbb129bce564a1e6c2f8afc
author Jeremy Bejarano <...> 1794324214 -0600
...
100644 blob dff2c50609d63181fc80dba80e5f685a78b4aea3	chartbook.lock
100644 blob 712f4033781c74ec6d75cf412b6e9b2b47344f9d	chartbook.toml
040000 tree ab75f65e5b03acf8a83ac63997dd3b7ed29690f5	data_manual
dff2c50609d63181fc80dba80e5f685a78b4aea3
dff2c50609d63181fc80dba80e5f685a78b4aea3  -

The last two lines are git’s hash of chartbook.lock and the same hash made by hand: SHA-1 of blob <size>\0 followed by the file. The lockfile’s own hash sits in the tree, the tree’s in the commit, the commit’s in its child.

The point: git’s one commit hash covers every file and all of history, a full Merkle tree. ChartBook stops a level short: a pinned_hash covers one dependency’s files, and a snapshot is named by its time and commit, not by a root hash. chartbook lineage checks the graph edge by edge instead. (chartbook lineage --rehash also reads every byte again, and chartbook store verify rehashes a whole store.)

D. A live git dependency#

What you are about to see: a public branch that grows by a row a day becomes a pinned upstream, with no snapshot made by its owner. The example is the Yield Spread Monitor from week 5, whose monitor-history branch holds spread_history.csv. For HW 4, your version of this step is your own HW 3 monitor; Make Your Homework an Upstream on the HW 4 page has the exact files for it.

That branch carries three small files beside the CSV, and they are what make it a ChartBook pipeline. (To add such files to a branch of your own, use GitHub’s web interface: switch to the branch, then Add file, committing to that branch and not a new one.) The daily workflow writes only spread_history.csv there, so the files stay. chartbook.toml:

[project]
name = "Yield Spread Monitor (history)"
description = "The 10-year minus 2-year Treasury yield spread, one row per business day since June 1976, recorded by the monitor's daily workflow."
maintainer = "Jeremiah Bejarano"
contributors = ["Jeremiah Bejarano"]
repo_url = "https://github.com/finm-32800/yield_spread_monitor"
os_compatibility = ["Windows", "Linux", "macOS"]

[dataframes.spread_history]
name = "10-Year Minus 2-Year Treasury Spread"
description = "Daily 10-year minus 2-year Treasury yield spread, in percentage points, from FRED's DGS10 and DGS2."
sources = ["FRED"]
providers = ["Federal Reserve Bank of St. Louis"]
provider_links = ["https://fred.stlouisfed.org/"]
tags = ["Rates", "Treasury"]
pull_method = "FRED API, src/pull_yields.py on main, run daily by GitHub Actions"
path = "spread_history.csv"
date_col = "date"
docs = "Two columns: date, and spread (DGS10 minus DGS2, in percentage points). The daily workflow on main appends one row each business day; nothing else writes this branch."

.gitattributes, one line, so that git on Windows does not rewrite the CSV’s line endings on checkout (which would change its bytes, and its hash):

* -text

And README.md:

# Yield Spread Monitor: history

The state the daily workflow carries from one day to the next, declared as a ChartBook dataframe. The code is on `main`.

To read it from another ChartBook pipeline, add this line to `[dependencies]` in its `chartbook.toml`, then run `chartbook pin`: ...

There is no [scripts] build: this branch builds nothing. Now, in the report, on a branch of your own, add the dependency and pin it:

git switch -c demo/yield-spread
echo '"finm-32800/yield_spread_monitor" = { git = "https://github.com/finm-32800/yield_spread_monitor", rev = "monitor-history" }' >> chartbook.toml
chartbook pin
git diff chartbook.lock
finm-32800/yield_spread_monitor: fetched https://github.com/finm-32800/yield_spread_monitor at aff8f6d (monitor-history)
finm-32800/yield_spread_monitor: 1 pinned
... 6 lines "already pinned"
Wrote chartbook.lock.
+          "id": "finm-32800/yield_spread_monitor",
+          "via": "rev",
+          "location": { "kind": "git", "url": "...", "rev": "monitor-history",
+            "commit": "aff8f6d5e6f2afa885ea6b431442702d2337fab9" },
+          "pinned_hash": "sha256:4385c35b8571..."
+      "id": "finm-32800/yield_spread_monitor:spread_history",
+      "location": { "kind": "file", "path": "spread_history.csv" },
+      "content_hash": "sha256:40a4699920c6...", "size_bytes": 202541

chartbook pin cloned the repository, read the files the branch’s chartbook.toml declares, copied them into your own store (the first one, which is why it must be writable), and recorded the commit and the hash. The six other dependencies say already pinned: nothing else moved, which is what a reviewer of your pull request checks. The key on the left of = is your name for the dependency, and references use it; use the ID ChartBook derives from the URL, the owner as it is and the repository name with - written _, so that the two agree. Read it, and commit:

python -c 'from chartbook import data
df = data.load("yield_spread_monitor:spread_history", format="pandas"); print(df.tail(3), len(df))'
git commit -qam "Depend on the yield spread monitor history"
chartbook lineage | grep spread
             date  spread
12603  2026-11-04    0.32
12604  2026-11-05    0.38
12605  2026-11-06    0.35 12606
├── finm-32800/yield_spread_monitor: pinned from its git clone at aff8f6d; not walked further

date comes back as strings, because the file is a CSV; an exhibit calls pd.to_datetime. Lineage stops at a git dependency: there is no snapshot to walk into.

Then the branch moves. The daily run adds a row at 22:00 UTC. (For this page, the branch was moved by editing its README instead, so that the commit changed and the data did not.) pin and update do different things:

chartbook pin | grep spread
chartbook update yield_spread_monitor
git diff chartbook.lock
finm-32800/yield_spread_monitor: 1 already pinned
chartbook.lock is unchanged.

finm-32800/yield_spread_monitor: fetched ... at f2180f9 (monitor-history)
-            "commit": "aff8f6d5e6f2afa885ea6b431442702d2337fab9"
+            "commit": "f2180f93cfa533511c420786e1060e1b3c3a5bc5"
             "pinned_hash": "sha256:4385c35b8571..."      (unchanged)

pin moved nothing. update fetched again, followed the branch to its new commit, and the diff shows exactly that: the commit line moved and no hash did, because the data did not change. data.path still points at the same stored file, so doit rebuilds no exhibit. After a day with a new row, the same two commands give a diff with a moved hash, of this shape:

finm-32800/yield_spread_monitor: 1 moved
  finm-32800/yield_spread_monitor:spread_history
    sha256:40a4699920c6… -> sha256:<the new file's>…
-            "commit": "f2180f93cfa533511c420786e1060e1b3c3a5bc5"
+            "commit": "<the day's commit>"
-          "pinned_hash": "sha256:4385c35b8571..."
+          "pinned_hash": "sha256:<new>..."
-      "size_bytes": 202541
+      "size_bytes": 202557

Sixteen more bytes: one row. The point: a pin never moves by itself. When it moves, the lockfile diff says what moved, and that diff is what you commit and what your reviewers read.

E. Rebuild versus read#

What you are about to see: the same code, run at a later point in time, gives a different hash, and the hash tells you so. Step B read a snapshot from a store. Now build the upstream yourself, at the commit the lockfile names. The output in this step is from November 2, when the report pinned commit 3904ad6 and a snapshot that ended October 2; your commit is the one your lockfile names (5bb7217 since November 3), and your dates are later. In a second terminal, so that the report’s environment stays active in the first:

git clone https://github.com/ftsfr/fed_yield_curve
cd fed_yield_curve
git switch --detach 5bb7217
python3.12 -m venv ../.venv && source ../.venv/bin/activate
pip install -r requirements.txt
chartbook trust
chartbook run --snapshot

(5bb7217 is the commit from step B; read it from the lockfile with jq -r '.records[0].dependencies[] | select(.id=="ftsfr/fed_yield_curve") | .location.commit' chartbook.lock in the report. Python 3.12, because the pipeline pins numpy==1.26.4; its requirements.txt pins pyarrow==20.0.0, which is what wrote the pinned file, and that matters at the end of this step. The environment goes beside the clone, not inside it: the pipeline’s .gitignore does not list .venv, and chartbook run --snapshot refuses a folder with uncommitted files.) chartbook trust says this folder’s build script may run on your machine. chartbook run --snapshot checks that everything is committed, builds (about a minute: it pulls the Fed’s yield curve file, formats it, runs the tests and two notebooks, and draws the chart), and copies what it built into your store:

ftsfr/fed_yield_curve: build
...
ftsfr/fed_yield_curve at 3904ad6: 4 asset(s) in the store
Snapshot made. A pipeline that depends on this one gets it with `chartbook update`.
Stores: default

Compare what you built with what the report pinned:

shasum -a 256 _data/ftsfr_treas_yield_curve_zero_coupon.parquet
bc106bbafd76e3ac33ee0b60bc0c8a1ccf96b09153d0773529853427766ec005  _data/ftsfr_treas_yield_curve_zero_coupon.parquet

Pinned b63da040..., rebuilt bc106bba.... Same commit, different bytes. The rebuild runs through October 23 (382,652 rows); the pin ends October 2 (382,232 rows). The rows they share agree exactly: the Fed published three more weeks and revised nothing. Back in the report, ChartBook sees the newer snapshot in your store and says so, without moving anything:

chartbook pin | grep -A5 fed_yield
ftsfr/fed_yield_curve: 4 changed upstream since it was pinned
  changed upstream: ftsfr/fed_yield_curve:ftsfr_treas_yield_curve_zero_coupon
  ... (chart, two notebooks)
4 pinned asset(s) have changed upstream. `chartbook update` moves the pins.

What you see depends on the day. If the Fed has published nothing since the pin, as on November 10, the rebuilt parquet is byte for byte the pinned one: chartbook pin then says 1 already pinned, 3 changed upstream (the chart and the two notebooks, which every build renders anew), and no exhibit rebuilds after an update.

To see what moving the pin does, move it, build, and undo:

chartbook update ftsfr/fed_yield_curve && git diff chartbook.lock | head -30
doit
git restore chartbook.lock && doit
ftsfr/fed_yield_curve: 4 moved
  ftsfr/fed_yield_curve:ftsfr_treas_yield_curve_zero_coupon
    sha256:b63da0404cf9… -> sha256:bc106bbafd76…
  From its snapshot at 3904ad6, made 2026-11-02 22:51 UTC, in store default.
-          "snapshot": "2026-11-02T05:22:19.659609Z",
+          "snapshot": "2026-11-02T22:51:43.146901Z",
-          "pinned_hash": "sha256:aff4ea7deb55..."
+          "pinned_hash": "sha256:1b172b12d4c4..."
-- exhibit:factor_returns  -- exhibit:gdp_growth  -- exhibit:size_value_returns
.  exhibit:yield_curve
.  compile_report  .  compile_style_guides  .  build_chartbook_site  .  run_pytest

The commit line did not move; the snapshot’s time and every hash did. Exactly the one exhibit that reads this pipeline rebuilt, and the PDF after it. Do not commit this in the shared report: the snapshot exists in your store only, so every classmate’s build, your reviewers’ included, would fail at chartbook pin --frozen. The HW 4 rule is that the pins in chartbook.lock are shared; if your exhibit needs the newer data, run chartbook update <name> and say so in your pull request, where the lockfile diff shows what moved.

Finally, does the rebuild reproduce the pin if you cut it at the pinned date (the last date the pin holds, which step B printed)? Save this as trim_and_hash.py and run it in the clone, with the clone’s environment:

"""Cut a rebuilt yield-curve file at the pinned date and hash what remains."""
import hashlib
import sys
from pathlib import Path

import pandas as pd
import pyarrow

src = Path(sys.argv[1])            # the rebuilt file
cutoff = sys.argv[2]               # the last date the pin holds
out = Path("trimmed.parquet")

df = pd.read_parquet(src)
df = df[df["ds"] <= cutoff]        # keep the index: the pipeline wrote it
df.to_parquet(out)
digest = hashlib.sha256(out.read_bytes()).hexdigest()
print(f"rows {len(df):,}, through {df['ds'].max():%Y-%m-%d}, "
      f"pyarrow {pyarrow.__version__}, pandas {pd.__version__}")
print(f"sha256:{digest}")
python trim_and_hash.py _data/ftsfr_treas_yield_curve_zero_coupon.parquet 2026-10-02
rows 382,232, through 2026-10-02, pyarrow 20.0.0, pandas 2.2.3
sha256:4c04214626a01006a131ff11fa391453a52c6f1dc15d5daede2d7704642a2506

Byte for byte the pin. The same rows written by pyarrow 26.0.0 hash to 62a8f97f5b01..., and by pyarrow 26.0.0 with pandas 3.1.0 to 5bb9af54f6d1...: same rows, three writers, three hashes.

The point: the hash pins the bytes, not the numbers. A rebuild reproduces a pin only with the same code, the same inputs, and the same environment, which is what uv.lock is for. When the hashes differ, the snapshot’s record says which commit made it and when, and that is where you start looking.

F. When do pins move#

Command

Pins

chartbook add <ID>, chartbook pin

Create a pin for a dependency that has none. Never move one; say changed upstream when a newer snapshot exists

chartbook run (and uv run doit in the report)

Never move one. The build reads the pinned snapshot. run also pins what has no pin yet, and builds a git dependency once

chartbook update [<name>]

Move: to the newest snapshot, or to what a branch means now. The lockfile diff is the record; commit it on its own

chartbook pin --frozen, chartbook run --frozen

Write nothing. Refuse if any pin is missing or not in a store. What CI runs

A git dependency with rev = "<commit>"

Cannot move: update fetches again and the commit means the same thing. A tag likewise; a branch moves

chartbook store gc

Keeps every snapshot a lockfile pins, the newest of each pipeline, and anything from the last seven days; removes the rest, after asking

For HW 4: chartbook pin after you add a line to [dependencies], read the diff, commit chartbook.lock, and leave the other pins where they are.