Python Packaging in Depth — pyproject.toml, the backend, the wheel

One file with three tables. PEP 517 for the build. PEP 518 for the config. PEP 621 for the metadata.

Build the wheel with the backend. Install it with pip. Develop against it with -e. Every packaging question reduces to one of the three surfaces.

§ IFrame

pyproject.toml is the file Python's packaging system reads to understand a project. It replaces setup.py, setup.cfg, requirements.txt, MANIFEST.in, and a scattered pile of tool-specific config files with one declarative source. Every modern Python project has one, most projects have one that grew organically over a decade of tooling changes, and very few developers can read one back and say what each section is for. This lesson names the pieces cold, so pyproject.toml becomes a legible artifact and packaging stops being ceremony.

The file is defined by three PEPs whose numbers are worth memorizing because they explain the layout. PEP 517 says packaging is done through a build-backend interface, not through calling setup.py directly. PEP 518 says the file that names the build-backend is pyproject.toml in the [build-system] table. PEP 621 says project metadata (name, version, dependencies, authors, entry points) goes in the [project] table of the same file, standardized across all backends. Together they define one file with two mandatory tables and a set of optional tool-specific ones, and every modern Python project should be readable by knowing that outline.

§ IILanguage Idiom

Three concepts sit under pyproject.toml, and knowing them in order clears the whole subject.

The build backend is what turns source into a distribution. When you run pip install . or python -m build, pip does not know how to build your package; it looks in pyproject.toml's [build-system] table for the build-backend field, invokes that backend's PEP 517 entry points, and receives a wheel back. The backend can be setuptools.build_meta (traditional, most compatible), hatchling.build (hatch's backend, clean modern option), poetry.core.masonry.api (poetry's), or flit_core.buildapi (minimal single-package option). Each backend reads the [project] table for standard metadata and its own table ([tool.hatch], [tool.poetry], and so on) for backend-specific options.

PEP 621 metadata is a stable contract. The [project] table's fields are the same regardless of which backend you use: name, version, description, readme, requires-python, dependencies, optional-dependencies, authors, license, keywords, classifiers, urls, scripts, gui-scripts, entry-points. A backend-swap is a mechanical exercise on [project] fields because they mean the same thing to every backend that reads them. When someone says "we're moving from poetry to hatch," they are typically changing [build-system] and adjusting one tool-specific table; the [project] section carries across unchanged.

Wheels and sdists are two artifact shapes. A wheel (.whl) is a pre-built binary distribution: the package's .py files (and any compiled .so files) laid out ready to extract into site-packages. pip install prefers wheels because installation is a unzip and a metadata check. An sdist (.tar.gz) is a source distribution: the full source tree plus pyproject.toml, meant to be built into a wheel on the target system. When you pip install a package from PyPI, pip downloads a wheel if one matches your Python version and platform; otherwise it downloads the sdist and builds a wheel locally. Publish both when you python -m build, because pure-Python packages install everywhere from the wheel, and platforms that need a locally-compiled extension fall back to the sdist.

§ IIICode Worked Example

The specimen is a complete pyproject.toml for the hedronite-ops package the Ops lesson named, using hatchling as the backend because it produces the smallest surface for a new package.

[build-system]
requires = ["hatchling>=1.24"]
build-backend = "hatchling.build"

[project]
name = "hedronite-ops"
version = "0.4.2"
description = "Shared fleet tools for the Hedronite ops team."
readme = "README.md"
requires-python = ">=3.11"
license = {text = "MIT"}
authors = [
  {name = "Hedronite Ops", email = "[email protected]"},
]
dependencies = [
  "kubernetes>=29.0",
  "httpx>=0.27",
  "tenacity>=8.4",
]

[project.optional-dependencies]
dev = ["pytest>=8.0", "ruff>=0.5", "mypy>=1.10"]

[project.scripts]
hedronite-sweep = "hedronite_ops.sweep:main"
hedronite-registry = "hedronite_ops.registry:cli"

[project.entry-points."hedronite_ops.plugins"]
retry-defaults = "hedronite_ops.retry:defaults"

[tool.hatch.build.targets.wheel]
packages = ["src/hedronite_ops"]

[tool.ruff]
target-version = "py311"
line-length = 100

Read it top-down. [build-system] names hatchling as the backend and pins a floor version because backend behavior changes across major versions; a version constraint here is the same discipline as version-pinning your runtime dependencies.

[project] is the metadata contract. name is the PyPI name (with hyphens); the importable package name (with underscores, hedronite_ops) is separate and lives in the source tree. version is a single source of truth here; some backends offer dynamic version via git tags or __version__ attribute, but the static version in pyproject.toml is the simplest shape and the exam-friendly one.

dependencies uses PEP 440 version constraints, same syntax as pip install kubernetes>=29.0. optional-dependencies groups extras: pip install hedronite-ops[dev] installs the dev group as well.

[project.scripts] maps CLI names to module:function targets. At install time, pip creates executable shims at the Python environment's bin/ directory. [project.entry-points."hedronite_ops.plugins"] registers this package as a plugin under the hedronite_ops.plugins group; other packages can discover its retry-defaults entry via importlib.metadata.entry_points(group="hedronite_ops.plugins").

The hatch-specific table [tool.hatch.build.targets.wheel] tells hatchling where the source lives; the src/ layout the Ops lesson recommended requires this pointer because hatch defaults assume the flat layout. Backend-specific tables are always under [tool.<backend-name>.*] by convention; anything under [tool.<name>] is claimed by the tool of that name and does not conflict with PEP 621.

§ IVBuilding, Installing, Editable Mode

Three commands cover the entire packaging lifecycle from source to distribution to install.

python -m build reads pyproject.toml, invokes the backend, and produces a wheel and an sdist in dist/. python -m build --wheel produces only a wheel; --sdist only an sdist. This is the artifact-creation step and the one that runs in CI to produce the files that get uploaded to PyPI.

pip install . also reads pyproject.toml, invokes the backend to build a wheel, and installs that wheel into the current environment. This is the developer-side equivalent of python -m build followed by pip install dist/*.whl, but as one step.

pip install -e . is the editable install, and it is what makes daily development possible. Instead of copying source into site-packages, pip creates a .pth file in site-packages that adds your project's source directory to sys.path. The upshot: your working copy is the installed package. Edit a .py file and the next import sees the change without reinstalling. Before PEP 660 (finalized 2021), editable installs were setuptools-specific and imperfectly supported; modern backends all implement PEP 660's build_editable hook and produce clean editable installs regardless of the backend.

Two failure modes deserve names. The ModuleNotFoundError immediately after an editable install almost always means the src/ layout is in use and the backend-specific config that points to it is missing. Hatchling needs packages = ["src/hedronite_ops"]; setuptools needs [tool.setuptools.packages.find] where = ["src"]; poetry needs packages = [{include = "hedronite_ops", from = "src"}]. Get the pointer wrong and the package is not found.

The other trap is the "installed package shadows source" case, where a stale pip install . remains in site-packages and imports resolve to the frozen copy rather than the working tree. Fix by uninstalling first: pip uninstall hedronite-ops then pip install -e .. Or use a fresh virtual environment, which is the discipline that avoids this class of problem entirely.

§ VPrior-Lesson Reach

The exception-model lesson from 07-24 lives in hedronite_ops.errors in this scheme; every custom exception hierarchy the fleet uses is defined once and imported everywhere. Sunday's structured-concurrency lesson lives in hedronite_ops.concurrency (or is a thin wrapper around asyncio primitives that gets re-exported). The context-manager governor from two weeks ago is one of the entry-point-registered plugins; another package can override the default budget by registering under hedronite_ops.plugins. The whole point of packaging is that these prior lessons stop being isolated ideas and become code someone else can import, extend, and depend on.

§ VIConnection to Today's Ops Lesson

The Ops lesson today named the shape; this lesson names the file that describes it to Python. Everything the Ops lesson recommended (src/ layout, entry-point CLI tools, plugin discovery via importlib.metadata.entry_points) becomes concrete here in a pyproject.toml that a backend can build. Read the two together and you have both altitudes: the operational shape and the metadata that produces it. The Cert lesson today lifts the same picture into the PCAP Modules and Packages objective, where the exam tests the import mechanics that packaging assumes and produces.

§ VIIClosing

pyproject.toml is one file with three obligations. The [build-system] table names the backend so pip knows how to build your package. The [project] table carries PEP 621 metadata that every backend reads the same way. A tool-specific [tool.<backend>] table (and [tool.ruff], [tool.mypy], etc.) carries the rest. Build with python -m build, install with pip install, develop with pip install -e ., and every packaging question reduces to which of the three surfaces you are editing.

Take a Python project you own that predates PEP 621. Rewrite its setup.py plus setup.cfg plus requirements.txt as one pyproject.toml with a modern backend of your choice. Run python -m build and pip install -e .. When the tool that appeared on your PATH is the tool that runs when you type its name, the packaging surface this lesson names has become code you own.