Modules, Packages, and if __name__ == "__main__"#

Every pipeline in this course is a folder of .py files that import one another. A guide notebook runs import pull_fed_funds_futures and calls load_fed_funds_futures(). dodo.py runs the same file as a script, with python ./src/pull_fed_funds_futures.py. And code that more than one repository needs, such as finm or chartbook, goes one step further: it becomes a package that each repository installs.

This notebook walks through the small examples in the python_imports/ folder of the in-class examples repository, from a single file to an installable package. Then it explains the if __name__ == "__main__": block at the bottom of nearly every module in this course.

Each example is a folder you can run yourself: cd into it and run python main.py. Here the notebook runs them and prints what they print.

import os
import subprocess
import sys
from pathlib import Path

HERE = Path.cwd()  # the python_imports/ folder, where this notebook runs


def show(path):
    """Print one of the example files, with its path as a header."""
    path = (HERE / path).resolve()
    print(f"--- {path.relative_to(HERE.parent)} ---")
    print(path.read_text().rstrip())


def run(script, folder):
    """Run `python <script>` inside `folder` and print its output.

    Paths in the output are shortened: `~` is your home folder and `<env>` is
    the folder of the Python environment the notebook runs in.
    """
    env = {**os.environ, "PYTHONDONTWRITEBYTECODE": "1"}
    result = subprocess.run(
        [sys.executable, script],
        cwd=HERE / folder,
        env=env,
        capture_output=True,
        text=True,
    )
    print(f"$ cd {folder} && python {script}")
    output = result.stdout + result.stderr
    output = output.replace(sys.prefix, "<env>").replace(str(Path.home()), "~")
    print(output.rstrip())

A module is a file#

Any .py file is a module. Another file in the same folder can import it by its name, without the .py:

show("01_simple_module/mypackage.py")
show("01_simple_module/main.py")
run("main.py", "01_simple_module")
--- python_imports/01_simple_module/mypackage.py ---
def say_hello():
    print("Hello from mypackage!")
--- python_imports/01_simple_module/main.py ---
"""
Importing a simple module.

A single .py file in the same directory can be imported by name.
"""

import mypackage

mypackage.say_hello()
$ cd 01_simple_module && python main.py
Hello from mypackage!

This is how the files in a course repository’s src/ folder use one another. Every module there starts with from settings import config: settings.py is a module in the same folder.

A package is a folder#

A folder that contains an __init__.py file is a package. It is imported the same way as a module, and the code in __init__.py runs when it is. A package can hold many modules, which are then imported as mypackage.something.

show("02_package_directory/mypackage/__init__.py")
show("02_package_directory/main.py")
run("main.py", "02_package_directory")
--- python_imports/02_package_directory/mypackage/__init__.py ---
def say_hello():
    print("Hello from mypackage!")
--- python_imports/02_package_directory/main.py ---
"""
Importing a package (directory with __init__.py).

A directory becomes a package when it contains an __init__.py file.
This is the difference between a module (single .py file) and a package (directory).
"""

import mypackage

mypackage.say_hello()
$ cd 02_package_directory && python main.py
Hello from mypackage!

Where Python looks#

import mypackage works because Python searches a list of folders, sys.path, in order, and the folder of the script you ran comes first. That is also how an import goes wrong. A file of your own named numpy.py hides the real NumPy:

show("03_name_shadowing/main.py")
run("main.py", "03_name_shadowing")
--- python_imports/03_name_shadowing/main.py ---
"""
Name shadowing: a local file can shadow an installed package.

If you create a file called numpy.py in your working directory,
`import numpy` will import YOUR file instead of the real numpy.
This is a common source of confusing errors.

Run this script and look at sys.path to understand why.
"""

import sys

print("sys.path (where Python looks for modules, in order):")
for p in sys.path:
    print(f"  {p}")

print()

# This imports the LOCAL numpy.py, not the real numpy!
import numpy

numpy.hello()
$ cd 03_name_shadowing && python main.py
sys.path (where Python looks for modules, in order):
  ~/GitRepositories/finm32800/inclass_examples/python_imports/03_name_shadowing
  <env>/lib/python312.zip
  <env>/lib/python3.12
  <env>/lib/python3.12/lib-dynload
  <env>/lib/python3.12/site-packages
  <env>/lib/python3.12/site-packages/aeosa

Hello from the LOCAL numpy.py, not the real numpy!

Two consequences for this course’s repositories:

  • python ./src/pull_fed_funds_futures.py can import settings, because src/ is the script’s own folder and so comes first on sys.path. dodo.py lives one folder up, so it puts src/ on the path itself: that is the sys.path.insert(1, "./src/") line at its top.

  • Never name a file after a package you use: no pandas.py, wrds.py, or doit.py in your project.

An installable package#

Code that more than one project needs should not be copied from one to the next. It becomes a package that each project installs. The smallest example is in project_paths/install_approach/: a package, src/myproj/, and a pyproject.toml that says what it is called, where its code is, and what it depends on.

show("../project_paths/install_approach/pyproject.toml")
--- project_paths/install_approach/pyproject.toml ---
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "myproj"
version = "0.1.0"
requires-python = ">=3.10"
dependencies = [
    "doit",
    "jupyter",
    "nbconvert",
]

[tool.hatch.build.targets.wheel]
packages = ["src/myproj"]

Each table in the file has one job. [build-system] names the tool that turns the source into an installable package, here Hatchling, and pip reads it to know how to build. [project] describes the package itself: its name, its version, the Python it needs, and the packages it depends on. [tool.hatch.build.targets.wheel] is a setting for Hatchling alone: it says the package’s code is in src/myproj.

pip install -e ., or uv pip install -e ., reads that file and puts myproj on sys.path for good, so import myproj works from any folder, in any script or notebook that uses the same environment. The -e, for editable, links to the source rather than copying it, so an edit to src/myproj/paths.py takes effect at the next import.

finm, chartbook, databento, and wrds are packages of exactly this shape, published to PyPI so that pip install can fetch them by name. Writing and Publishing Your Own Python Packages takes it from here.

if __name__ == "__main__"#

Every module has a variable called __name__. When the module is imported, __name__ is the module’s name, "greet" for a file called greet.py. When the file is the one you run, as in python greet.py, Python sets __name__ to "__main__" instead. So a block under if __name__ == "__main__": runs when the file is run as a script, and is skipped when the file is imported.

show("04_name_main/greet.py")
run("greet.py", "04_name_main")
--- python_imports/04_name_main/greet.py ---
"""A module that is also a script.

Run it directly (`python greet.py`) and the block at the bottom runs. Import it
(`import greet`, as main.py does) and the block is skipped.
"""

print(f"Loading greet.py; __name__ is {__name__!r}")


def say_hello():
    print("Hello from greet!")


if __name__ == "__main__":
    # Runs only when this file is the one you ran: python greet.py
    say_hello()
$ cd 04_name_main && python greet.py
Loading greet.py; __name__ is '__main__'
Hello from greet!
show("04_name_main/main.py")
run("main.py", "04_name_main")
--- python_imports/04_name_main/main.py ---
"""Import greet.py as a module.

The top-level print in greet.py runs on import. The block under
`if __name__ == "__main__":` does not, so main.py calls say_hello() itself.
"""

import greet

print("main.py imported greet, and now calls greet.say_hello():")
greet.say_hello()
$ cd 04_name_main && python main.py
Loading greet.py; __name__ is 'greet'
main.py imported greet, and now calls greet.say_hello():
Hello from greet!

The print at the top of greet.py runs both times, because everything outside the block runs on import. The block itself runs only when greet.py is the file you ran.

Why the course’s modules end this way#

A pull_ module in this course does two jobs. As a script, it pulls data from a vendor and saves it, and that is how dodo.py runs it. As a module, it offers a load_ function that reads what the pull saved, and that is how a notebook uses it. Here is the shape of the FedWatch case study’s pull_fed_funds_futures.py, shortened:

def pull_fed_funds_futures(start_date=START_DATE, end_date=END_DATE):
    """Pull daily ZQ bars from Databento."""      # goes out to the network
    ...


def load_fed_funds_futures(data_dir=DATA_DIR):
    """Load the cached ZQ daily bars pulled by this module."""
    return pd.read_parquet(Path(data_dir) / PARQUET_NAME)


if __name__ == "__main__":
    df = pull_fed_funds_futures()
    df.to_parquet(DATA_DIR / PARQUET_NAME)

dodo.py runs python ./src/pull_fed_funds_futures.py, so __name__ is "__main__" and the pull runs. The guide notebook runs import pull_fed_funds_futures and calls load_fed_funds_futures(), so the block is skipped and nothing touches the network. Without the guard, every import of the module, by a notebook, a test, or another module, would pull the data again.

The block has a second common use: a quick check of a module by hand. paths.py in the install example prints the paths it computes when you run it, and prints nothing when it is imported:

run("src/myproj/paths.py", "../project_paths/install_approach")
$ cd ../project_paths/install_approach && python src/myproj/paths.py
Project root: ~/GitRepositories/finm32800/inclass_examples/project_paths/install_approach
Data dir:     ~/GitRepositories/finm32800/inclass_examples/project_paths/install_approach/_data
Output dir:   ~/GitRepositories/finm32800/inclass_examples/project_paths/install_approach/_output

What to remember#

  • A file is a module, and a folder with an __init__.py is a package. sys.path decides which one an import finds.

  • An installed package can be imported from anywhere, which is how code is shared between projects.

  • Code under if __name__ == "__main__": runs when the file is run, not when it is imported. In this course, that block is where a pull_ module does its pulling.