Study Guide: Week 8#

Open-ended questions on the material from week 8. The exam asks multiple-choice versions of questions like these. No answers are given: each question can be answered in a few sentences from the pages listed under its heading. The exam’s format is described in Exam Preparation.

Pins That a Rebuild Can Meet#

Read: Pins That a Rebuild Can Meet.

  1. In HW 4 the report’s lockfile recorded a hash of the bytes of each upstream file. A classmate checked out the same upstream commit a week later, rebuilt it, and got a table whose hash did not match. No code had changed. What had?

  2. Why can no rebuild reproduce the bytes of a Plotly HTML chart or of an executed notebook, even from the same code and the same data?

  3. A pin by bytes works when a team shares one store, and fails when each person rebuilds the upstream. Why?

  4. Suppose a table is pinned by a hash of its rows up to the pin’s last date, and not by the bytes of its file. For each change below, say whether the pin is still met, and why.

    • The source has published two new days of data.

    • The same rows were written in another order, by another version of the Parquet writer.

    • The source revised a value from last year.

Corporate Bonds in March 2020#

Read: Corporate Bonds in March 2020.

  1. What is a credit spread? When a bond’s credit spread widens and Treasury yields do not move, what has happened to the bond’s price?

  2. What happened to investment-grade and high-yield credit spreads between late February and March 20, 2020?

  3. What did the Federal Reserve announce on March 23, 2020? Its first purchases came weeks later. What does the timing of the fall in spreads say about how the announcement worked?

  4. The figure of daily spreads and trade counts is drawn from cleaned TRACE. Name one way the same figure drawn from the raw feed would be wrong.

  5. In most crises investors buy Treasuries, and Treasury yields fall. In the dash for cash of March 2020 they sold Treasuries too. Why does that point to a need for cash rather than fear of default, and how does it fit with the safest corporate bonds’ spreads rising the most in proportion?

Port Forwarding and VS Code#

Read: Remote Machines and High-Performance Computing.

  1. VS Code is running on a compute node of Midway3 and listening on a port. Why can the browser on your laptop not open it directly?

  2. In the command below, what does each of 8000, midway3-0123, and 41234 name? Which machine looks up the name midway3-0123?

    ssh -L 8000:midway3-0123:41234 cnetid@midway3.rcc.uchicago.edu
    
  3. With that command running, you open http://localhost:8000 in a browser. Trace the request from the browser to VS Code, machine by machine.

  4. scode starts VS Code inside a SLURM job. What happens to your editor when the job’s time limit runs out, and what happens to the files you saved?

Parallel Jobs: GNU Parallel and Job Arrays#

Read: Parallel Jobs on a Cluster.

  1. What makes a set of tasks independent enough to run in parallel? Stage 0 of the TRACE pipeline can be split by month. Why?

  2. What does the command below run, and how many of them at a time?

    parallel -j 2 python clean_month.py {} ::: 2020-02 2020-03 2020-04
    
  3. What does GNU Parallel write to the file named by --joblog, and what does --resume do with it on the next run?

  4. How many jobs does the script below create, and what does each one run?

    #SBATCH --array=0-2
    MONTHS=(2020-02 2020-03 2020-04)
    python clean_month.py ${MONTHS[$SLURM_ARRAY_TASK_ID]}
    
  5. After submitting that script as job 12345, sacct -j 12345 shows 12345_1 as FAILED and the other two as COMPLETED. What command reruns only the task that failed?

  6. A task that exits at once when its output file already exists is called idempotent. How does that make rerunning a whole array cheap?

  7. What does the %10 do in #SBATCH --array=0-99%10?

  8. The scheduler sees a job array as many jobs and a GNU Parallel run as one job. What does that difference mean for when the work starts and for how you find out which task failed?

  9. When is one job that runs GNU Parallel a better choice than a job array?