Projects

A useful-views project is a folder with a fixed structure. Every project looks the same, so a teammate, a reviewer or an AI tool knows where everything is.

my-data-project/
├── README.md          # written by you: the project and its data sources
├── AGENTS.md          # instructions for AI coding assistants; written once by init
├── datapackage.json   # generated: what each recipe made from what
├── cookbook.json      # generated by useful-views cookbook: the recipes used
├── .gitignore         # written once by init
├── raw/               # raw inputs, placed by hand; never written by a recipe
├── recipes/           # one folder per recipe, written by you
├── work/              # in-between files, one folder per recipe
└── useful-views/      # the declared views, plus the generated catalog

Create one with useful-views init my-data-project, or start from a ready-made example with useful-views example pwt.

What lives where

Kind Where Made by In git
Raw input raw/ (subfolders allowed) You, by hand No
Work file work/<recipe>/ A recipe’s steps, for its own later steps No
Useful view useful-views/, flat A recipe, as a declared output No (parquet); the catalog yes
Record datapackage.json, useful-views/README.md useful-views, after each successful run Yes
Cookbook cookbook.json useful-views cookbook, from the record Yes
  • Raw inputs are the data you start from. Nothing a recipe does writes to raw/. Because raw/ is not committed, say in the project’s README.md where each file came from.
  • Work files pass data from one step to the next within a recipe. No other recipe may read them, and run empties work/<recipe>/ before each run, so a leftover never passes for a fresh result.
  • Useful views are the derived tables you actually use. Each is a .parquet file made by exactly one recipe, and another recipe may read it as an input.
  • Records say which recipe made each view, from which inputs, with which steps. See Records.

Who writes what

You write the project README.md and everything in recipes/; init and new-recipe only start them from templates and never touch them again. useful-views writes datapackage.json, useful-views/README.md and cookbook.json; never edit those by hand. init writes .gitignore and AGENTS.md once, if they are not there; after that they are yours.

What to commit

Commit README.md, AGENTS.md, recipes/, datapackage.json, cookbook.json, .gitignore, raw/.gitkeep and useful-views/README.md, and the files that list your steps’ libraries (pyproject.toml and uv.lock, requirements.txt or environment.yml; see Installation). The generated .gitignore keeps the rest out:

raw/*
!raw/.gitkeep
/work/
useful-views/*.parquet
__pycache__/

If the folder already had a .gitignore, init leaves it alone and warns if git would not ignore raw data, work files or views, naming the lines to add.

A teammate who clones the project installs the libraries, places the raw files as your README.md describes, then runs useful-views sync, which makes every view again.

Sharing raw data

raw/ is not in git, because raw data is often large, private or licensed. Share it the way its license allows: the original source (with the exact download date, in your README.md), your institution’s storage, or a data repository. The record in datapackage.json lists every raw file each recipe read, with its columns, so a teammate can check they have the right files.

Finding the project from a subfolder

Every command and function finds the project by walking up from the current folder to datapackage.json, so they work from a notebook in notebooks/. From anywhere else, name it: useful-views -C path/to/project check, or root= in Python.

init refuses to make a project in your home folder or at the top of a drive, since every folder below would then be inside it (--force if you really mean it). Projects cannot be nested.