Study Guide: Week 7#
Open-ended questions on the material from week 7. The exam asks multiple-choice versions of questions like these. No answers are given: each question can be answered in a few sentences from the pages listed under its heading. The exam’s format is described in Exam Preparation.
Issues and Pull Requests#
Read: GitHub Issues and Pull Requests.
What is a pull request? What is the difference between pushing a commit to
mainand opening a pull request?git pulland a pull request sound alike. What does each one do?What is a GitHub issue, and what does writing
fixes #123in a pull request do?Why do maintainers protect
mainand require a review before merging?
Cleaning TRACE#
Read: Data Sources Overview and Cleaning TRACE Corporate Bond Data.
What does TRACE record, and why does its raw feed contain cancellations, corrections, and reversals?
What is an agency double count, and what goes wrong in a volume or liquidity measure if it is left in?
Two careful researchers clean the same TRACE feed and get different datasets. Why is that possible, and what makes either result defensible?
Medium Data and Polars#
Read: Medium-Sized Data Strategies and Polars Exercises.
Name the three ways out when data does not fit in memory. Which one solves most problems?
What is the difference between eager and lazy evaluation in Polars?
What do predicate pushdown and projection pushdown each skip reading in the query below?
(pl.scan_parquet("trace/*.parquet") .filter(pl.col("trd_exctn_dt") >= date(2020, 1, 1)) .select(["cusip_id", "rptd_pr"]) .collect())
A dataset is stored as
trace/year=2019/...,trace/year=2020/..., and so on. What is this layout called, and what does a filter onyearsave?What does
collect(engine="streaming")orsink_parquetbuy you thatcollect()does not?
Remote Machines#
Read: Remote Machines and High-Performance Computing.
What is the difference between a login node and a compute node? Why must you not run a long computation on a login node?
On Midway3, what belongs in home, in project, and in scratch?
Most compute nodes have no internet access. How does the TRACE pipeline still pull from WRDS, and what does separating pulls from processing buy?
What does each of
-a,-v, and-zdo inrsync -avz, and why isrsyncbetter thanscpfor a directory you sync repeatedly?What does
ssh -N -f -L 8888:10.50.1.2:8888 midway3do?What do
ControlMaster,ControlPath, andControlPersistin an SSH config buy you on a cluster that asks for Duo?scode, Remote-SSH, andcode tunnelall put VS Code on a remote machine. Where does the server run in each, and why doescode tunnelnot work on a Midway3 compute node?
SLURM and Grid Engine#
Read: Remote Machines and High-Performance Computing.
What does a job scheduler do, in four steps?
What is the difference between
sinteractiveandsbatch?What does each line of this script ask for?
#SBATCH --partition=caslake #SBATCH --time=00:02:00 #SBATCH --mem=1G #SBATCH --array=1-5 #SBATCH --output=%x_%A_%a.out
When is a set of tasks a good fit for a job array, and when is it not?
What do
squeue,scancel, andsaccteach tell you or do?Give the Grid Engine command on the WRDS Cloud that matches each of
sinteractive,sbatch,squeue, andscancel.