03. The Daily Monitor#
In this guide
A monitor is a replication that runs itself. It adds five things to the pipeline: a schedule, a secret, a locked environment, a ledger, and a deploy.
The history is the one output kept between runs. Every other output is rebuilt from source; the history would cost a pull per day to rebuild, so it lives on a branch of its own, one small commit a day.
An outage is not a bug. A data source that is down ends the run with a warning and leaves yesterday’s page up; any other failure fails the run.
Guide 02 computed SVIX and the probability of a crash for one session.
This page is what happens when a replication is run every day without
anyone at the keyboard. A GitHub Actions workflow, .github/workflows/deploy_pages.yml,
wakes up at 14:30 UTC, pulls the previous session’s settlements from
Databento, computes the row, appends it to the history, records the
history on a branch of its own, rebuilds the monitor’s page, and publishes
it. The chart below, the one on that page, is one point longer every
morning.
Sections
The history, and the latest row.
What the workflow does, step by step, and the places it can go wrong.
Why the history lives on its own branch.
import pandas as pd
from IPython.display import HTML
import figures # noqa: F401 sets the notebook renderer for Plotly
import monitor
import monitor_chart
pd.set_option("display.width", 140)
pd.set_option("display.max_columns", 30)
history = monitor.load_history()
print(f"{len(history)} sessions, {history['date'].min().date()} to {history['date'].max().date()}")
35 sessions, 2026-09-14 to 2026-10-30
The history#
One row per session. The columns are documented in src/monitor.py; the
ones the chart uses are the 30-day SVIX (svix_30), the VIX close for
the same day from FRED, and the three probabilities of a 20% drop over
the expiration closest to one month: risk-neutral (p_star_20), the log
investor’s (p_log_20), and the power-utility investor’s with
\(\gamma = 2\) (p_power_20).
cols = ["date", "days_used", "svix_30", "vix", "p_star_20", "p_log_20", "p_power_20", "p_star_10", "p_log_10"]
history[cols].tail(10).set_index("date").style.format({
"svix_30": "{:.2f}", "vix": "{:.2f}",
"p_star_20": "{:.2%}", "p_log_20": "{:.2%}", "p_power_20": "{:.2%}", "p_star_10": "{:.2%}", "p_log_10": "{:.2%}",
})
| days_used | svix_30 | vix | p_star_20 | p_log_20 | p_power_20 | p_star_10 | p_log_10 | |
|---|---|---|---|---|---|---|---|---|
| date | ||||||||
| 2026-10-19 00:00:00 | 42 | 16.64 | 18.44 | 1.01% | 0.68% | 0.47% | 3.70% | 2.99% |
| 2026-10-20 00:00:00 | 41 | 15.89 | 17.31 | 0.80% | 0.54% | 0.37% | 3.31% | 2.70% |
| 2026-10-21 00:00:00 | 40 | 15.37 | 16.68 | 0.67% | 0.45% | 0.31% | 3.19% | 2.63% |
| 2026-10-22 00:00:00 | 39 | 17.29 | 19.03 | 1.00% | 0.68% | 0.47% | 3.73% | 3.03% |
| 2026-10-23 00:00:00 | 38 | 17.84 | 19.45 | 0.98% | 0.66% | 0.45% | 4.09% | 3.34% |
| 2026-10-26 00:00:00 | 35 | 16.61 | 18.39 | 0.80% | 0.55% | 0.38% | 3.04% | 2.46% |
| 2026-10-27 00:00:00 | 34 | 15.94 | 17.52 | 0.70% | 0.48% | 0.34% | 2.81% | 2.30% |
| 2026-10-28 00:00:00 | 33 | 15.39 | 16.91 | 0.59% | 0.41% | 0.28% | 2.68% | 2.22% |
| 2026-10-29 00:00:00 | 32 | 14.97 | 17.20 | 0.53% | 0.37% | 0.25% | 2.52% | 2.09% |
| 2026-10-30 00:00:00 | 31 | 17.95 | 19.42 | 0.80% | 0.55% | 0.38% | 3.54% | 2.92% |
monitor_chart.plot_plotly(history).show()
This is the chart the monitor publishes, drawn by src/monitor_chart.py
from the history. src/build_page.py puts it on a single HTML page, with
the latest reading and the last ten sessions, and that page is the whole
published site. Hover for any day’s values.
Takeaway. Both panels move with the smile of guide 01, and the risk-neutral probability is always the largest of the three. SVIX sits a little below the VIX. Both are annualized volatilities read off the same options, but the VIX weights each strike by \(1/K^2\) and SVIX weights them equally, so the VIX leans harder on the put wing; the gap is a measure of how steep the smirk is. The probabilities move with the smile too, and the risk-neutral one is always the largest: it charges for crash states at their marginal utility.
What the workflow does#
uv run doit latest is the whole update. The task pulls the latest
session (cheap: 19 MB, $0.00, a minute or two), computes its row if the
history does not have it, appends it, and rebuilds the page from the
history, _output/monitor_page/index.html. Run it on your laptop and it
does the same thing; open the page in a browser to see it. Plain uv run doit is the same
computation over the window committed in src/settings.py (START_DATE
to END_DATE), so it is also the backfill that gives the chart a history
behind it: move START_DATE earlier, commit, and the next run fills the
history; on Actions a second workflow does the same, started from the
Actions tab. The daily workflow adds the parts that make the update
unattended:
A schedule. A
cronline underon: schedule:starts the run every morning, after CME’s final settlements of the previous evening have reached Databento. GitHub’s cron runs in UTC; the five fields are minute, hour, day of month, month, and day of week.A secret. The pull needs your Databento key. It lives in the repository’s secret store (Settings → Secrets and variables → Actions), and the step that needs it reads it from there through the
secretscontext. It is never in the file, never in a commit, never in the log.An environment.
astral-sh/setup-uvinstallsuv;uv run doitthen installs Python 3.12 and the locked dependencies fromuv.lock, the same versions as on your laptop.A ledger. The updated history is committed to the
monitor-historybranch from a second working directory (git worktree): one small commit a day, the day’s row. A push to that branch does not start the workflow, which listens to pushes onmainonly.A deploy. The page is force-pushed as a single orphan commit on the
gh-pagesbranch, which GitHub Pages serves. Only the page: the notebooks and the ChartBook site stay on your laptop.
Three places it goes wrong, in our experience. A workflow that commits
its output to main retriggers itself on every run, forever, unless each
push says [skip ci]; keeping output off main removes the problem. A
deploy that does not git clean -fdx first carries _data/ and .venv/
into the published branch. And a data source can be down: Databento and FRED
both time out now and then. The pulls retry server and network errors,
waiting longer each time, for about eight minutes. If a source is still
down after that, the step after the pull sees the marker the pull left
and ends the run with a warning, not a failure: yesterday’s page stays
up, and the next morning’s pull, which covers the last four calendar days,
fills the gap.
Any other failure is a bug and fails the run.
Stop and discuss
Databento is down for three days. What does the run look like on each of those mornings, what does the published chart show, and what happens on the fourth morning?
Why the history lives on its own branch#
Everything else the pipeline produces is regenerated from source: pull
again, compute again, same answer. The history is different. Each row
costs a session’s pull, and a year of them would cost a year of pulls, so
the monitor keeps what it has computed. Not on main, though: main
holds source code, and nothing a run produces belongs there. The workflow
keeps the one file, svix_monitor_history.csv, on a branch of its own,
monitor-history. Each morning it restores the file from there into
_output/, appends the new sessions, and commits the result back as one
small commit, so the branch is the ledger of the series, one line a day,
and git log -p on it shows the chart being built point by point. The
page itself sits on gh-pages as a single commit that each run replaces.
The daily run adds one session. The history behind it comes from a
backfill, the same computation over a range of dates, run once by hand;
if the engine ever changes, the same backfill rebuilds the history.
Stop and discuss
The runner starts empty every morning. Where does yesterday’s row come
from? Why not keep the history on main, on gh-pages, or in the Actions
cache?
latest = history.iloc[-1]
HTML(
f"<p>As of <b>{latest['date']:%A, %B %d, %Y}</b>: SVIX {latest['svix_30']:.1f}, VIX {latest['vix']:.1f}; "
f"the options market prices a 20% drop over the next {int(latest['days_used'])} days at "
f"<b>{100 * latest['p_star_20']:.2f}%</b> risk-neutral, which a log investor who holds the market reads as "
f"<b>{100 * latest['p_log_20']:.2f}%</b>.</p>"
)
As of Friday, October 30, 2026: SVIX 17.9, VIX 19.4; the options market prices a 20% drop over the next 31 days at 0.80% risk-neutral, which a log investor who holds the market reads as 0.55%.
Notes on the discussion questions
Three days without Databento. Each of those mornings the pull retries
for about eight minutes, leaves the SOURCE_UNAVAILABLE marker, and the
run ends with a warning: no row is recorded and the page is not
redeployed, so the published chart stays as it was, short of the latest
sessions. On the fourth morning the daily pull covers the last four
calendar days, so on weekdays it picks up all three missed sessions and the
history is whole again. A longer outage, or one that spans a weekend,
leaves sessions outside that window; the backfill workflow, run once after
a commit moves the window in src/settings.py over the gap, fills them.
Where yesterday’s row comes from. From the monitor-history branch:
the restore step fetches it and copies the CSV into _output/ before the
update runs. Not main: it holds source code, and a workflow that pushed
output to main would trigger itself on every run. Not gh-pages: each
run replaces it with one fresh commit, so anything kept there is wiped
daily. Not the Actions cache: GitHub evicts a cache that goes unused for a
week, and a cache has no history, so losing it means paying for every pull
again. A branch is durable, free, and versioned: git log -p on it is the
series, one row a day.