Modules, Packages, and if __name__ == "__main__"#
Every pipeline in this course is a folder of .py files that import one
another. A guide notebook runs import pull_fed_funds_futures and calls
load_fed_funds_futures(). dodo.py runs the same file as a script, with
python ./src/pull_fed_funds_futures.py. And code that more than one
repository needs, such as finm or chartbook, goes one step further: it
becomes a package that each repository installs.
This notebook walks through the small examples in the
python_imports/
folder of the in-class examples repository, from a single file to an
installable package. Then it explains the if __name__ == "__main__": block
at the bottom of nearly every module in this course.
Each example is a folder you can run yourself: cd into it and run
python main.py. Here the notebook runs them and prints what they print.
import os
import subprocess
import sys
from pathlib import Path
HERE = Path.cwd() # the python_imports/ folder, where this notebook runs
def show(path):
"""Print one of the example files, with its path as a header."""
path = (HERE / path).resolve()
print(f"--- {path.relative_to(HERE.parent)} ---")
print(path.read_text().rstrip())
def run(script, folder):
"""Run `python <script>` inside `folder` and print its output.
Paths in the output are shortened: `~` is your home folder and `<env>` is
the folder of the Python environment the notebook runs in.
"""
env = {**os.environ, "PYTHONDONTWRITEBYTECODE": "1"}
result = subprocess.run(
[sys.executable, script],
cwd=HERE / folder,
env=env,
capture_output=True,
text=True,
)
print(f"$ cd {folder} && python {script}")
output = result.stdout + result.stderr
output = output.replace(sys.prefix, "<env>").replace(str(Path.home()), "~")
print(output.rstrip())
A module is a file#
Any .py file is a module. Another file in the same folder can import
it by its name, without the .py:
show("01_simple_module/mypackage.py")
show("01_simple_module/main.py")
run("main.py", "01_simple_module")
--- python_imports/01_simple_module/mypackage.py ---
def say_hello():
print("Hello from mypackage!")
--- python_imports/01_simple_module/main.py ---
"""
Importing a simple module.
A single .py file in the same directory can be imported by name.
"""
import mypackage
mypackage.say_hello()
$ cd 01_simple_module && python main.py
Hello from mypackage!
This is how the files in a course repository’s src/ folder use one
another. Every module there starts with from settings import config:
settings.py is a module in the same folder.
A package is a folder#
A folder that contains an __init__.py file is a package. It is
imported the same way as a module, and the code in __init__.py runs when
it is. A package can hold many modules, which are then imported as
mypackage.something.
show("02_package_directory/mypackage/__init__.py")
show("02_package_directory/main.py")
run("main.py", "02_package_directory")
--- python_imports/02_package_directory/mypackage/__init__.py ---
def say_hello():
print("Hello from mypackage!")
--- python_imports/02_package_directory/main.py ---
"""
Importing a package (directory with __init__.py).
A directory becomes a package when it contains an __init__.py file.
This is the difference between a module (single .py file) and a package (directory).
"""
import mypackage
mypackage.say_hello()
$ cd 02_package_directory && python main.py
Hello from mypackage!
Where Python looks#
import mypackage works because Python searches a list of folders,
sys.path, in order, and the folder of the script you ran comes first.
That is also how an import goes wrong. A file of your own named numpy.py
hides the real NumPy:
show("03_name_shadowing/main.py")
run("main.py", "03_name_shadowing")
--- python_imports/03_name_shadowing/main.py ---
"""
Name shadowing: a local file can shadow an installed package.
If you create a file called numpy.py in your working directory,
`import numpy` will import YOUR file instead of the real numpy.
This is a common source of confusing errors.
Run this script and look at sys.path to understand why.
"""
import sys
print("sys.path (where Python looks for modules, in order):")
for p in sys.path:
print(f" {p}")
print()
# This imports the LOCAL numpy.py, not the real numpy!
import numpy
numpy.hello()
$ cd 03_name_shadowing && python main.py
sys.path (where Python looks for modules, in order):
~/GitRepositories/finm32800/inclass_examples/python_imports/03_name_shadowing
<env>/lib/python312.zip
<env>/lib/python3.12
<env>/lib/python3.12/lib-dynload
<env>/lib/python3.12/site-packages
<env>/lib/python3.12/site-packages/aeosa
Hello from the LOCAL numpy.py, not the real numpy!
Two consequences for this course’s repositories:
python ./src/pull_fed_funds_futures.pycanimport settings, becausesrc/is the script’s own folder and so comes first onsys.path.dodo.pylives one folder up, so it putssrc/on the path itself: that is thesys.path.insert(1, "./src/")line at its top.Never name a file after a package you use: no
pandas.py,wrds.py, ordoit.pyin your project.
An installable package#
Code that more than one project needs should not be copied from one to the
next. It becomes a package that each project installs. The smallest example
is in
project_paths/install_approach/:
a package, src/myproj/, and a pyproject.toml that says what it is called,
where its code is, and what it depends on.
show("../project_paths/install_approach/pyproject.toml")
--- project_paths/install_approach/pyproject.toml ---
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[project]
name = "myproj"
version = "0.1.0"
requires-python = ">=3.10"
dependencies = [
"doit",
"jupyter",
"nbconvert",
]
[tool.hatch.build.targets.wheel]
packages = ["src/myproj"]
Each table in the file has one job. [build-system] names the tool that
turns the source into an installable package, here Hatchling, and pip
reads it to know how to build. [project] describes the package itself: its
name, its version, the Python it needs, and the packages it depends on.
[tool.hatch.build.targets.wheel] is a setting for Hatchling alone: it says
the package’s code is in src/myproj.
pip install -e ., or uv pip install -e ., reads that file and puts
myproj on sys.path for good, so import myproj works from any folder,
in any script or notebook that uses the same environment. The -e, for
editable, links to the source rather than copying it, so an edit to
src/myproj/paths.py takes effect at the next import.
finm, chartbook, databento, and wrds are packages of exactly this
shape, published to PyPI so that pip install can fetch them by name.
Writing and Publishing Your Own Python Packages
takes it from here.
if __name__ == "__main__"#
Every module has a variable called __name__. When the module is imported,
__name__ is the module’s name, "greet" for a file called greet.py. When
the file is the one you run, as in python greet.py, Python sets __name__
to "__main__" instead. So a block under if __name__ == "__main__": runs
when the file is run as a script, and is skipped when the file is imported.
show("04_name_main/greet.py")
run("greet.py", "04_name_main")
--- python_imports/04_name_main/greet.py ---
"""A module that is also a script.
Run it directly (`python greet.py`) and the block at the bottom runs. Import it
(`import greet`, as main.py does) and the block is skipped.
"""
print(f"Loading greet.py; __name__ is {__name__!r}")
def say_hello():
print("Hello from greet!")
if __name__ == "__main__":
# Runs only when this file is the one you ran: python greet.py
say_hello()
$ cd 04_name_main && python greet.py
Loading greet.py; __name__ is '__main__'
Hello from greet!
show("04_name_main/main.py")
run("main.py", "04_name_main")
--- python_imports/04_name_main/main.py ---
"""Import greet.py as a module.
The top-level print in greet.py runs on import. The block under
`if __name__ == "__main__":` does not, so main.py calls say_hello() itself.
"""
import greet
print("main.py imported greet, and now calls greet.say_hello():")
greet.say_hello()
$ cd 04_name_main && python main.py
Loading greet.py; __name__ is 'greet'
main.py imported greet, and now calls greet.say_hello():
Hello from greet!
The print at the top of greet.py runs both times, because everything
outside the block runs on import. The block itself runs only when
greet.py is the file you ran.
Why the course’s modules end this way#
A pull_ module in this course does two jobs. As a script, it pulls data
from a vendor and saves it, and that is how dodo.py runs it. As a module,
it offers a load_ function that reads what the pull saved, and that is how
a notebook uses it. Here is the shape of the FedWatch case study’s
pull_fed_funds_futures.py, shortened:
def pull_fed_funds_futures(start_date=START_DATE, end_date=END_DATE):
"""Pull daily ZQ bars from Databento.""" # goes out to the network
...
def load_fed_funds_futures(data_dir=DATA_DIR):
"""Load the cached ZQ daily bars pulled by this module."""
return pd.read_parquet(Path(data_dir) / PARQUET_NAME)
if __name__ == "__main__":
df = pull_fed_funds_futures()
df.to_parquet(DATA_DIR / PARQUET_NAME)
dodo.py runs python ./src/pull_fed_funds_futures.py, so __name__ is
"__main__" and the pull runs. The guide notebook runs
import pull_fed_funds_futures and calls load_fed_funds_futures(), so the
block is skipped and nothing touches the network. Without the guard, every
import of the module, by a notebook, a test, or another module, would pull
the data again.
The block has a second common use: a quick check of a module by hand.
paths.py in the install example prints the paths it computes when you run
it, and prints nothing when it is imported:
run("src/myproj/paths.py", "../project_paths/install_approach")
$ cd ../project_paths/install_approach && python src/myproj/paths.py
Project root: ~/GitRepositories/finm32800/inclass_examples/project_paths/install_approach
Data dir: ~/GitRepositories/finm32800/inclass_examples/project_paths/install_approach/_data
Output dir: ~/GitRepositories/finm32800/inclass_examples/project_paths/install_approach/_output
What to remember#
A file is a module, and a folder with an
__init__.pyis a package.sys.pathdecides which one animportfinds.An installed package can be imported from anywhere, which is how code is shared between projects.
Code under
if __name__ == "__main__":runs when the file is run, not when it is imported. In this course, that block is where apull_module does its pulling.