Documented derived tables for data projects.

Links

View on PyPI
Issues

AI / Agents

Skills
llms.txt
llms-full.txt

Developers

James Horine

Community

Full license MIT
Citing useful-views

Meta

Requires: Python >=3.11
Package Info

Overview

useful-views is a Python package that gives a data project a fixed structure, so that every derived table (a “useful view”) records the recipe that made it. Every recipe records the steps that make that recipe.

Full documentation, including a command reference and a guide for AI tools: https://jameshorine.github.io/useful-views/

Installation

uv init --bare             # once, if the folder has no pyproject.toml yet
uv add useful-views polars # or, in an activated virtual environment: pip install useful-views polars

Install it into the same Python environment as the libraries your steps use (polars, pandas, …): steps run with the Python that runs useful-views. So do not run it with uvx or pipx. The installation guide covers pip, conda, Windows and notebooks.

Usage

With the environment active (or with uv run in front of each command):

useful-views example pwt      # a ready-made example; for your own data: useful-views init my-project
cd pwt-example
useful-views sync             # run every recipe, upstream first, and record what each made
useful-views check            # report anything missing, refused or out of date
useful-views cookbook         # write cookbook.json, the recipes the project uses

The quickstart walks through it, in Python and on the command line.

The resulting data project structure

Using useful-views in a data project results in this layout:

my-data-project/
├── README.md                     # written by humans; documents the data project
├── AGENTS.md                     # written once by init: instructions for AI coding assistants
├── datapackage.json              # generated: resources and recipes (never hand-edited)
├── cookbook.json                 # generated by useful-views cookbook: the recipes used, in run order
├── .gitignore                    # generated by init: ignores raw/, work/, and other data files
│
├── raw/                          # raw inputs: git-ignored, never written by a recipe
│   ├── .gitkeep                  # keeps the folder in git
│   └── my_data.extension         # any format
│
├── recipes/                      # written by humans, AI tools, or both
│   ├── recipe_01/                # one folder = one recipe, at least 1 step, maybe more!
│   │   ├── recipe.toml           # description, inputs, outputs
│   │   ├── 01_first_step.py      # one file = at least one step; top comment is its description
│   │   ├── 02_second_step.py
│   │   └── 03_third_step.py
│   │
│   └── recipe_02/                # a second recipe, taking a view as its input
│       ├── recipe.toml           # inputs = ["useful-views/recipe-01.parquet"]
│       └── 01_only_step.py       # smallest valid recipe: a file containing a single step
│
├── work/                         # intermediates, never recorded, git-ignored
│   ├── recipe_01/                # one scratch folder per recipe, emptied before each run
│   │   ├── step_1.extension      # any format
│   │   └── step_2.extension
│   └── recipe_02/
│
└── useful-views/                 # declared outputs only, flat
    ├── README.md                 # generated: catalog of the views and how each was made
    ├── recipe-01.parquet         # produced by recipes/recipe_01
    └── recipe-02.parquet         # produced by recipes/recipe_02

Raw inputs and work files can be in any format. Views are parquet files: parquet keeps each column’s type, so the record in datapackage.json and the catalog can say exactly what every view contains. Other view formats may be allowed in the future.

Vocabulary

Using useful-views is meant to be easy! However, we have a few (hopefully) useful vocabulary terms to help our collective practice and understanding.

The building blocks are raw inputs, steps and recipes.

Term	What it is	Where it lives
Raw input	A file you start from. Nothing in raw/ is produced by a recipe.	raw/
Step	One file that does at least one non-trivial action on the data.	A file in a recipe folder
Recipe	Like cooking: steps run in order. From the inputs it declares, it produces the useful views it declares.	A folder in recipes/

For example, in polars, given:

df = pl.DataFrame(
    {
        "foo": [-20.5, float("nan"), 80.0],
        "bar": [float("nan"), 110.0, 25.5],
        "ham": ["xxx", "yyy", None],
    }
)

calling polars.DataFrame.drop_nans() returns:

>>> df.drop_nans()
shape: (1, 3)
┌──────┬──────┬──────┐
│ foo  ┆ bar  ┆ ham  │
│ ---  ┆ ---  ┆ ---  │
│ f64  ┆ f64  ┆ str  │
╞══════╪══════╪══════╡
│ 80.0 ┆ 25.5 ┆ null │
└──────┴──────┴──────┘

This would be considered a step. So, again, a step is at least one non-trivial transformation action on the data. That is the minimum. There is no maximum yet: a step may do many transformations, and the author decides where one step ends and the next begins.

Work: intermediate files, in work/<recipe>/, read only by later steps of the same recipe.

Useful view: a declared output of a recipe, in useful-views/. Every view is produced by exactly one recipe. A view can be an input to another recipe. Anything read from outside the recipe that made it must be a view, not a work file.

Writing a recipe

A recipe folder holds one recipe.toml and at least one step file.

recipes/pwt_panel/recipe.toml:

description = "Country-year panel of core Penn World Table variables for Brazil, Germany, Spain and the United States."
inputs  = ["raw/pwt110_bra_deu_esp_usa.csv"]
outputs = ["useful-views/pwt-panel.parquet"]

recipes/pwt_panel/01_reshape_long.py:

# Reshape the wide export (one column per year) into one row per country, variable and year.
import polars as pl

wide = pl.read_csv("raw/pwt110_bra_deu_esp_usa.csv")
long = wide.unpivot(
    index=["ISO code", "Country", "Variable code", "Variable name"],
    variable_name="year",
    value_name="value",
).with_columns(pl.col("year").cast(pl.Int32))
long.write_parquet("work/pwt_panel/long.parquet")

The conventions:

• Inputs and outputs are written as paths from the project root. The path in recipe.toml is the same path the step uses in its code.
• An input is a file under raw/, or a view under useful-views/ made by another recipe.
• An output is always a file directly inside useful-views/.
• Step files are named NN_name.py and run in numeric order.
• The comment at the top of a step file is its description. It is required.
• A step is an ordinary script. It runs from the project root and does not import useful_views. Use whichever data library you like.
• Steps pass data to each other only through files in work/<recipe>/.

In detail:

• Recipe folder name: lowercase letters, digits and underscores, such as pwt_panel. Create one with useful-views new-recipe <name>; its template says TODO and is refused until you fill it in.
• recipe.toml has exactly three keys. description is one sentence, required. inputs and outputs are lists of at least one path each. Other keys are ignored with a warning.
• Inputs: a file anywhere under raw/, or a view directly inside useful-views/ that another recipe makes and has already made successfully.
• Outputs: one or more .parquet files directly inside useful-views/. Each view is made by exactly one recipe.
• Paths: relative to the project root, with forward slashes, matching the letter case on disk. Write raw/data.csv, not ./raw/data.csv, raw\data.csv or an absolute path. The error message gives the corrected string where it can; inside double quotes a backslash is a TOML escape, so then it only tells you to use forward slashes.
• Step files: NN_name.py, two digits, an underscore and a name. They run in numeric order; 00_ and gaps in the numbering are fine. A file named almost like a step, such as 1_clean.py, 01-clean.py, 01 clean.py or 01_clean.py.txt, is not run, and run warns about it. R, notebook, Quarto, Julia and Stata files named like a step (02_clean.R, 02_explore.ipynb, …) are refused: steps must be Python files. Any other file in the folder is ignored, unless it starts with NN_ and ends in .py (then it is a step: keep old copies elsewhere).
• Step description: the block of # lines at the very top of the file, its lines joined with spaces into one line. Blank lines, a shebang, a coding line, notebook cell markers (# %%, # In[1]:, with any title on the same line), a Jupytext # --- header, an inline # /// script block and decoration lines like # ----- before it are skipped; a docstring is not, so put the comment first. It must not start with TODO. Step files are UTF-8.
• Steps run one at a time as separate processes, with the project root as the working directory, using the same Python interpreter as useful-views, and with work/<recipe>/ already created and empty. A step gets no keyboard input, so input() and breakpoint() fail inside run; to debug a step, run it yourself from the project root with the same Python, as in python -X utf8 recipes/pwt_panel/01_reshape_long.py, after the recipe has run once (so work/<recipe>/ exists), then run the recipe again to record it. A step reads its recipe’s inputs and earlier steps’ files in work/<recipe>/, and writes the recipe’s outputs. It never writes to raw/, and never reads another recipe’s work files.

Example data

Example projects live in a separate repository, useful-views-data, the way seaborn’s example datasets live in seaborn-data. useful-views example <name> [path] (Python: example(name, path)) downloads one into a new folder, ready to run with useful-views sync. Each release of useful-views pins a version of that repository and the checksum of every file, so an example never changes under you and a damaged download is never used.

The pwt example uses an extract of the Penn World Table, version 11.0, for Brazil, Germany, Spain and the United States. The Penn World Table is licensed under a Creative Commons Attribution 4.0 International License; the example’s README.md gives the exact source and the citation.

Commands

Python	Command line	What it does
init(path=".")	useful-views init [path]	Create the structure above. Safe to run twice; never overwrites a file.
example(name, path=None)	useful-views example <name> [path]	Download a ready-to-run example project, such as pwt. See Example data.
new_recipe(name)	useful-views new-recipe <name>	Create a recipe folder with a template recipe.toml and one stub step.
run(recipe)	useful-views run <recipe>	Run one recipe’s steps in order and record what it produced.
sync(all=False)	useful-views sync [--all]	Make the views match the recipe folders, like uv sync: run what is missing or changed, upstream first, and remove what is no longer declared.
check()	useful-views check	Report everything run and sync would refuse, every recipe sync would run, and what is missing or out of place. Changes nothing.
cookbook(check=False)	useful-views cookbook [--check]	Write cookbook.json, the ledger of the recipes used, in run order. --check verifies it, for CI.
RULES	useful-views rules	List the numbered rules a recipe must follow.
root()	none	Return the project root, for building paths in a notebook.

run and sync differ the way uv run and uv sync do: run does the one recipe you name, now; sync brings the whole project in line with its recipe folders.

What run does

Before changing anything on disk, run checks the recipe and how it fits the project (see Rules). Then it:

1. deletes the recipe’s declared views and empties work/<recipe>/, so a leftover file can never pass for a fresh result;
2. runs each step in order. Their output appears as they run, in a terminal or a notebook cell. The first step that fails stops the run;
3. checks that every declared view was written as a non-empty parquet file;
4. records the run: it rewrites datapackage.json and useful-views/README.md. Only this recipe’s entries change; every other recipe’s record is kept exactly as it was.

If a step fails or a view is missing, nothing is recorded: the recipe’s views are removed, and its record from the last successful run, if any, is kept. check then reports the views as missing, or the recipe as never run. The files in work/<recipe>/ stay as the steps left them, so you can see how far the run got. Running the same recipe twice leaves datapackage.json and the catalog byte-for-byte unchanged.

run never runs other recipes. If an input is another recipe’s view, run that recipe first, or use sync. If other recipes read the views run just made again, it warns and names them: their views are now out of date.

What sync does

sync makes the project match its recipe folders, the way uv sync makes an environment match its lock file. Records only ever come from real runs.

1. It checks every recipe against the rules, judging each against the records as they will be when its turn comes. If any recipe would be refused, it lists every problem at once and changes nothing. A folder with no recipe.toml is skipped with a warning.
2. It removes the records of recipes whose folder was deleted, and any view a recipe no longer lists in its outputs, along with those view files. It warns about recipes that read a removed view.
3. It runs, upstream first, every recipe that has never run, whose recipe.toml or steps no longer match its record, or whose views are missing, and everything downstream of those. --all (all=True) runs every recipe. It stops at the first failure or interruption; recipes that ran before it stay recorded, and the message says what to run next.

On a fresh clone, place the raw files and run useful-views sync to make every view. sync does not notice a change inside a step’s code, a raw file replaced under the same name, or a git pull or branch switch (the views are not in git). After any of those, use sync --all, or run the changed recipe and then each recipe downstream of it.

Only one command that changes a project runs at a time: a second run, sync or cookbook started meanwhile is refused instead of overwriting the first one’s records.

Steps are code

run and sync execute every step in recipes/ as you, with your permissions, your environment variables (including any passwords or tokens in them) and your network access. There is no sandbox. Read the recipes of a project you did not write, or changes someone proposes, before running them. check, rules and cookbook never run a step.

What check reports

Kind	Meaning
unreadable-recipe	A recipe folder whose recipe.toml is missing or cannot be read.
refused	A recipe breaks a rule, so run and sync would refuse it. The finding names the rule.
missing-input	An input a recipe.toml declares is not on disk.
never-run	A recipe that has never run successfully.
out-of-date	A recipe sync would run again: its recipe.toml or steps changed, its outputs changed, or a view it reads will be made again.
missing-recipe-folder	datapackage.json records a recipe whose folder is gone. sync removes the record.
missing-view	A recorded view whose file is missing.
stale-cookbook	cookbook.json no longer matches the recorded recipes. Update it with useful-views cookbook.
stale-catalog	useful-views/README.md does not match datapackage.json (edited by hand, or left from a merge). sync writes it again.
stray-file	A file or folder in useful-views/, other than README.md, that no recipe makes.

Each finding says what to do about it, and check uses the same plan as sync, so a clean check means sync has nothing to refuse. It does not notice changes inside a step’s code or to a file’s contents.

Rules

run refuses a recipe, before changing anything on disk, when:

1. recipe.toml is missing or unreadable, lacks description, inputs or outputs, has an empty list, lists a path twice, or its description starts with TODO.
2. The recipe has no step file; a step has no top comment, or one that starts with TODO; or two steps share a number.
3. A file in the recipe folder is named like a step but is an R, notebook, Quarto, Julia or Stata file (.R, .Rmd, .ipynb, .qmd, .jl, .do): steps must be .py files.
4. An input is not a file under raw/ or directly inside useful-views/, is written in the wrong form (or as a wildcard), has a name that cannot exist on Windows, does not exist with exactly that letter case, or is an empty or unreadable raw file.
5. An input inside useful-views/ is not made by another recipe that has already run successfully.
6. The recipe would create a cycle: it reads its own view, directly or through a chain of other recipes.
7. An output is not a .parquet file directly inside useful-views/, has a name that cannot exist on Windows, is useful-views/README.md, or is already made by another recipe.
8. Recording it would give two files the same resource name (a view’s file name, or a raw file’s path below raw/, without the extension, lowercased, with / and any other ASCII character than a-z, 0-9, ., _ or - turned into -; accented letters lose their accents, and letters of other scripts become their Unicode code point), such as raw/data.csv and useful-views/data.parquet, or raw/My Data.csv and useful-views/my-data.parquet.

run also refuses a recipe name that does not match a folder in recipes/, or a folder name that is not allowed (lowercase letters, digits and underscores, and not a name Windows reserves, such as con or aux). Every refusal names the rule it broke and says what to fix, and in which file. useful-views rules lists the rules; from Python they are useful_views.RULES. A refusal raises RecipeRefused, whose rule is the number above (or None for the name refusals), rule_title its title, and path the file to fix.

Using it from Python

import useful_views as uvs

uvs.run("pwt_panel")                 # works from any folder inside the project
uvs.check(root="~/data/my-project")  # or name the project folder explicitly
gdp = uvs.root() / "useful-views" / "gdp-per-capita.parquet"

• new_recipe, run, sync, check, cookbook and root find the project by walking up from the current folder to datapackage.json, so they work from a notebook in notebooks/. All of them but root also take root=: the project folder itself, the one holding datapackage.json. init and example take the folder to create.
• init and example return the project folder, new_recipe the recipe folder, run a RunResult (recipe, outputs, warnings), sync a SyncResult (removed, dropped, deleted, ran, reasons, warnings), cookbook a CookbookResult (path, written, up_to_date, recipes, findings), and check a list of findings (kind, path, message, rule, recipe, reason).
• Failures raise subclasses of UsefulViewsError: RecipeRefused (rule, rule_title, path, refusals), StepFailed (step, returncode, stderr), OutputError (paths, problems), CookbookError (findings), ExampleError (kind), ProjectNotFound and ProjectError. Each also has recipe, sync_result and to_dict(). Warnings are UsefulViewsWarning.
• On the command line, the exit code is 0 for success, 1 when check finds problems or cookbook --check finds the cookbook out of date, 2 for a usage error, 3 when an operation is refused or fails, 130 when interrupted and 143 when terminated. useful-views --version prints the version. python -m useful_views ... works the same as useful-views ..., for example when the useful-views command is not found.

For scripts and AI tools

• --json on any command, --help and --version included, prints exactly one JSON document on standard output, also on errors, with any warnings in it; everything else goes to standard error.
• -C <path> runs a command in another folder, like git -C.
• Every error carries its facts as attributes and to_dict(): the recipe, the rule, the file to fix, and the failed step and its error output.
• init writes an AGENTS.md into each new project, with the rules an AI coding assistant must follow there.
• In CI, useful-views cookbook --check needs no data: it checks that cookbook.json and datapackage.json match the recipe folders.

See Working with AI tools.

Generated files

datapackage.json is a Data Package that records what each recipe made from what. frictionless validate datapackage.json passes on it whenever every recorded file is present.

• resources lists every raw file a recorded recipe read, then every view, each with its columns and their types. A view also has producedBy: the recipe that made it and that recipe’s inputs.
• recipes lists each recorded recipe with its description, inputs, outputs and steps, as they were at its last successful run.
• Column types come from the files themselves: a parquet file’s own types, and for a raw CSV, types inferred from the whole file. In a raw CSV only an empty cell counts as missing, so a column that marks missing values with NA or . is recorded as string. Other raw formats (.dta, .xlsx, …), and CSVs that are not plain comma-separated UTF-8, are recorded without columns.
• Never edit it by hand. If it is damaged, restore it from git.

useful-views/README.md is a catalog generated from datapackage.json: for each view, the recipe’s description, its inputs, its numbered steps, and its columns.

cookbook.json is the ledger of the recipes the project uses, written by useful-views cookbook the way uv lock writes uv.lock: every recorded recipe in run order, upstream first, with its description, inputs, outputs and steps, and the raw inputs the recipes read. To write it, the project must be in sync. useful-views cookbook --check exits 1 when it is missing or out of date, or when a recipe changed without a sync. It needs no data, so CI can require an up-to-date cookbook. See Cookbook for what it holds and a ready-made CI job.

.gitignore is written once by init, if there is none:

raw/*
!raw/.gitkeep
/work/
useful-views/*.parquet
__pycache__/

So raw data, work files and views stay out of git, while recipes/, datapackage.json, cookbook.json, AGENTS.md and the catalog are committed, with the files that list your steps’ libraries (pyproject.toml and uv.lock, or requirements.txt). A teammate installs those, places the raw files and runs useful-views sync to make the views again.

Who writes what

Humans write the project README.md and everything in recipes/; init and new-recipe only start them from templates and never touch them again. The package writes datapackage.json, useful-views/README.md and, when you run useful-views cookbook, cookbook.json; never edit those by hand. init writes .gitignore and AGENTS.md once, if they are not there; after that they are yours. So no file has two authors at the same time.

Planned, TODO, Not yet

• Hashing. Version 1 of cookbook.json records every recipe and its steps. Later versions will add the SHA-256 of every raw input, recipe file, step and view, and the environment the steps ran in, locking them the way uv.lock locks a Python environment. Then useful-views can report a view as stale, and sync can notice a change inside a step’s code, a replaced raw file, or views left from before a git pull.
• Whether a step should have an upper limit on how much it does.
• Recording where a raw file came from and how to fetch it again.
• More view formats alongside parquet, such as CSV and Arrow.
• Raw inputs that are folders or sets of files, such as partitioned parquet datasets and shapefiles.
• Subfolders inside useful-views/, and notebooks as steps.

License

useful-views is released under the MIT License; see LICENSE. The example data is not part of the package and not under its license: it lives in the useful-views-data repository (with a copy in tests/data/ for the tests), and is an extract of the Penn World Table, licensed CC BY 4.0. The example’s README.md gives the citation.

Citing useful-views

If you use useful-views in your research, please cite it. GitHub’s Cite this repository button (from CITATION.cff) gives APA and BibTeX, or use:

Horine, J. (2026). useful-views: Documented derived tables for data projects (Version 0.1.0) [Computer software]. https://github.com/jameshorine/useful-views

@software{horine_useful_views_2026,
  author  = {Horine, James},
  title   = {useful-views: Documented derived tables for data projects},
  year    = {2026},
  version = {0.1.0},
  url     = {https://github.com/jameshorine/useful-views},
  license = {MIT}
}

Please also cite the sources of any data you use with it; the pwt example uses the Penn World Table, whose citation is in the example’s README.md.