----------------------------------------------------------------------
This is the API documentation for the useful_views library.
----------------------------------------------------------------------


## Functions

Create a project, run its recipes, and report its state.


init(path: 'str | Path' = '.', *, force: 'bool' = False) -> 'Path'

Create the project structure at `path`.

Creates `raw/`, `recipes/`, `work/` and `useful-views/`, a starter
`README.md`, an `AGENTS.md` with instructions for AI coding assistants, a
`.gitignore` that keeps raw data, work files and views out of git, and an
empty `datapackage.json` and catalog. Safe to run twice: it never
overwrites a file. If a `.gitignore` is already there, it warns when raw
data, work files or views would not be ignored (asking git, inside a git
repository), naming the lines to add.

Args:
    path: The project folder. Created, with its parents, if missing.
    force: Create the project even in your home folder or at the top of a
        drive, which init otherwise refuses: every folder below would then
        be inside the project.

Returns:
    The project folder, as an absolute path.

Raises:
    ProjectError: `path` is inside another project, is a file, holds a
        `datapackage.json` that useful-views did not write, or is your home
        folder or the top of a drive (without `force`).

example(name: 'str', path: 'str | Path | None' = None) -> 'Path'

Download a ready-to-run example project.

The example's raw data and recipes come from the useful-views-data
repository, at the version this release of useful-views pins, and each
file is checked against its SHA-256 before anything is written. The
folder is then set up like `init` does. Make its views with
`useful-views sync`.

Args:
    name: The example, such as `"pwt"` (Penn World Table extract).
    path: The folder to create; default `"<name>-example"`. It must not
        exist, or be empty.

Returns:
    The project folder, as an absolute path.

Raises:
    ExampleError: The name is unknown, the folder is not empty (or is
        your home folder or the top of a drive), a file could not be
        downloaded, or a checksum did not match. Nothing is written in
        those cases.
    ProjectError: The folder is inside another project.

new_recipe(name: 'str', *, root: 'str | Path | None' = None) -> 'Path'

Create `recipes/<name>/` with a template `recipe.toml` and a stub step.

Both files say TODO and explain what to fill in. Until you replace the
TODOs, `run` and `sync` refuse the recipe (rules 1 and 2), and `check`
reports it as refused.

Args:
    name: Lowercase letters, digits and underscores, like `"pwt_panel"`,
        and not a name Windows reserves (con, aux, nul, prn, com1...) or
        `__pycache__`.
    root: The project folder. By default it is found by walking up from
        the current folder to `datapackage.json`.

Returns:
    The new recipe folder.

Raises:
    RecipeRefused: The name is not allowed, or the folder already exists.
    ProjectNotFound: No project was found.

run(recipe: 'str', *, root: 'str | Path | None' = None) -> 'RunResult'

Run one recipe's steps in order and record what it produced.

Before anything on disk changes, the recipe is checked against the rules
(`useful-views rules`). Then its views are deleted and `work/<recipe>/`
is emptied, each step runs as its own process (from the project root,
with this Python, its output shown as it runs), every declared view must
have been written, and the run is recorded in `datapackage.json` and
`useful-views/README.md`. Other recipes' records are not changed, and no
other recipe is run: if other recipes read this one's views, a warning
names them, because their views were made from the old ones (run
`useful-views sync --all`, or run each of them, upstream first).

Args:
    recipe: The recipe's name, as in `"pwt_panel"` or `"recipes/pwt_panel/"`.
    root: The project folder. By default it is found by walking up from
        the current folder to `datapackage.json`.

Returns:
    A `RunResult` naming the views written.

Raises:
    RecipeRefused: The recipe breaks a rule; nothing changed.
    StepFailed: A step failed; nothing was recorded and the recipe's views
        were removed.
    OutputError: A declared view was missing, empty or unreadable after
        the steps ran; nothing was recorded.
    ProjectError: Another useful-views command is changing the project,
        or a file could not be written or deleted.
    ProjectNotFound: No project was found.

sync(*, root: 'str | Path | None' = None, all: 'bool' = False) -> 'SyncResult'

Make the views and datapackage.json match the recipe folders, like `uv sync`.

First, every recipe folder is checked against the rules; if any recipe
would be refused, sync raises before changing anything, listing every
problem. Then it removes the records and view files of recipes whose
folder is gone and of outputs a recipe no longer declares, and runs,
upstream first, every recipe that has never run, whose recipe folder no
longer matches its record, or whose views are missing, plus everything
downstream of those. Progress lines go to standard error; steps' own
output appears as they run.

If a recipe fails, or sync is interrupted, sync stops there. What it had
already done is kept, and the views of recipes it had not reached that
were made from views it had already made again are removed, so the next
sync makes them again.

Args:
    root: The project folder. By default it is found by walking up from
        the current folder to `datapackage.json`.
    all: Run every recipe, not only those that are missing or changed.
        Use it after editing the code inside a step, which sync does not
        notice by itself, and after pulling or switching branches.

Returns:
    A `SyncResult` saying what was removed, deleted and run.

Raises:
    RecipeRefused: A recipe breaks a rule; nothing changed. Its
        `refusals` lists every problem found.
    StepFailed, OutputError: A recipe failed while running. The removals
        and the recipes that ran before it are kept; the exception's
        `sync_result` says what was done, and its message what to run next.
    ProjectError: Another useful-views command is changing the project,
        or a file could not be written or deleted.
    ProjectNotFound: No project was found.

check(*, root: 'str | Path | None' = None) -> 'list[Finding]'

Report the state of the project, without changing anything.

Reports what is missing or out of place, every rule a recipe breaks that
would stop `run` or `sync` (`refused`), and recipes that `sync` would run
(`never-run`, `out-of-date`, `missing-view`). It does not notice changes
inside a step's code or to a file's contents.

Args:
    root: The project folder. By default it is found by walking up from
        the current folder to `datapackage.json`.

Returns:
    A list of `Finding`s in a fixed order; empty when there is nothing to
    report.

cookbook(*, root: 'str | Path | None' = None, check: 'bool' = False) -> 'CookbookResult'

Write `cookbook.json`, the ledger of the recipes the project uses.

The cookbook lists every recorded recipe, in run order (upstream first),
with its description, inputs, outputs and steps, and the raw inputs the
recipes read. It is generated from the record in `datapackage.json`, so
to write it the project must be in sync: if `check` finds anything other
than stray files or a stale catalog or cookbook, nothing is written. Writing an unchanged cookbook
leaves the file untouched. Commit it to git, like a lock file.

`check=True` needs no data: it works on a fresh clone or in CI, where
raw/ and the views are not present. It compares `cookbook.json` with the
committed `datapackage.json`, and also reports recipes whose folder no
longer matches their record (edited, added or removed without a sync).

Args:
    root: The project folder. By default it is found by walking up from
        the current folder to `datapackage.json`.
    check: Do not write; report whether `cookbook.json` matches the
        project's recipes. Use it in CI.

Returns:
    A `CookbookResult`. With `check=True`, its `up_to_date` is the
    answer, and its `findings` list recipes whose record is out of date.

Raises:
    CookbookError: When writing, the project is not in sync; its
        `findings` say why.
    ProjectError: cookbook.json exists but cannot be read.
    ProjectNotFound: No project was found.

root() -> 'Path'

The project folder, found from the current folder.

Walks up from the current folder to the nearest `datapackage.json` that
useful-views wrote, so it works from a notebook in `notebooks/`. Use it
to build paths to views and read them with any library:

    gdp = pl.read_parquet(uvs.root() / "useful-views" / "gdp-per-capita.parquet")

Returns:
    The project folder, as an absolute path.

Raises:
    ProjectNotFound: The current folder is not inside a project.


## Results

What run, sync, check and cookbook return.


RunResult(recipe: 'str', outputs: 'tuple[str, ...]', warnings: 'tuple[str, ...]') -> None

What `run` did.

Attributes:
    recipe: The recipe that ran.
    outputs: The views it wrote, as paths from the project root.
    warnings: Warnings issued along the way (also shown as they happen).

SyncResult(removed: 'tuple[str, ...]', dropped: 'tuple[str, ...]', deleted: 'tuple[str, ...]', ran: 'tuple[str, ...]', warnings: 'tuple[str, ...]', reasons: 'dict[str, str]' = <factory>) -> None

What `sync` did.

Attributes:
    removed: Recipes whose records were removed because their folder is gone.
    dropped: Views removed from a record because the recipe no longer declares them.
    deleted: Files deleted: views no longer declared, and the work/
        folders of the removed recipes (ending in "/").
    ran: Recipes run, in the order they ran.
    warnings: Warnings issued along the way (also shown as they happen).
    reasons: Why each recipe in `ran` ran: "never-run", "changed" (its
        recipe.toml or steps no longer matched its record),
        "outputs-changed", "views-missing", "upstream" (a view it reads
        was made again) or "all" (`all=True`).

Finding(kind: 'str', path: 'str', message: 'str', rule: 'int | None' = None, recipe: 'str | None' = None, reason: 'str | None' = None) -> None

One problem `check` found.

Attributes:
    kind: What kind of problem: one of "unreadable-recipe", "refused",
        "missing-input", "never-run", "out-of-date", "missing-recipe-folder",
        "missing-view", "stale-cookbook", "stale-catalog" or "stray-file".
        The check findings reference lists what each means.
    path: The file or folder concerned, relative to the project root.
    message: What is wrong and what to do about it.
    rule: For `refused`, the number of the broken rule; otherwise None.
    recipe: The recipe concerned, when there is one.
    reason: For `out-of-date`, why: "changed" (its recipe.toml or steps
        no longer match its record), "outputs-changed" (it no longer
        declares any view it made), "outputs-dropped" (it no longer
        declares some of them; sync removes them without running it) or
        "upstream" (a view it reads will be made again). For
        `missing-input`, "made-by-recipe" when the input is a view
        another recipe makes, which sync makes first. Otherwise None.

CookbookResult(path: 'Path', written: 'bool', up_to_date: 'bool', recipes: 'tuple[str, ...]' = (), findings: 'tuple' = ()) -> None

What `cookbook` did, or found with `check=True`.

Attributes:
    path: The cookbook file, `cookbook.json` at the project root.
    written: True if the file was created or changed.
    up_to_date: True if the file on disk matches the project's recipes
        (after writing, always True; with `check=True`, the answer).
    recipes: The recipes in the cookbook, in run order.
    findings: With `check=True`, the `check` findings that mean the
        recorded recipes no longer match the recipe folders (so the
        cookbook cannot be up to date); otherwise empty.


## Errors and warnings

Every error is a UsefulViewsError and carries its facts as attributes and to_dict().


UsefulViewsError(message: 'str', *, recipe: 'str | None' = None)

Base class for every error useful-views raises.

Attributes:
    recipe: The recipe the error is about, when there is one.
    sync_result: When the error stopped `sync`, a `SyncResult` saying
        what sync had already done (those changes are kept); else None.

RecipeRefused(message: 'str', *, rule: 'int | None' = None, path: 'str | None' = None, recipe: 'str | None' = None)

A recipe was refused because it breaks a rule; nothing on disk changed.

Attributes:
    rule: The number of the broken rule (see `useful_views.RULES` or
        `useful-views rules`), or None when the recipe name itself is
        refused.
    rule_title: The rule's one-line title, or None.
    path: The file to fix, relative to the project root, when there is one.
    recipe: The recipe that was refused.
    refusals: Every refusal found at once. `sync` checks all recipes
        before changing anything and raises the first, with the rest here;
        otherwise this is just `[self]`.

StepFailed(message: 'str', *, step: 'str', returncode: 'int', stderr: 'str', recipe: 'str | None' = None)

A step exited with an error, so nothing was recorded.

The recipe's views were removed; its work files are kept for inspection.

Attributes:
    step: The step file, relative to the project root.
    returncode: The step's exit code; negative when a signal stopped it
        (on Linux and macOS: -9 is SIGKILL, often from running out of
        memory).
    stderr: What the step wrote to its error output: all of it, or its
        last 64 KB, with a note saying how much came before.
    recipe: The recipe the step belongs to.

OutputError(message: 'str', *, paths: 'list[str]', problems: 'list[tuple[str, str]] | None' = None, recipe: 'str | None' = None)

After the steps ran, a declared output was missing, empty or unreadable.

Nothing was recorded, and the recipe's views were removed.

Attributes:
    paths: The outputs that were not right.
    problems: (path, reason) for each of them.
    recipe: The recipe whose outputs these are.

ProjectError(message: 'str', *, recipe: 'str | None' = None)

The project itself is missing or damaged, or a file could not be written.

ProjectNotFound(message: 'str', *, recipe: 'str | None' = None)

No useful-views project was found from the current folder or `root=`.

ExampleError(message: 'str', *, kind: 'str | None' = None)

An example could not be downloaded or set up.

Attributes:
    kind: What went wrong, for scripts: "unknown-example", "folder-not-empty",
        "network", "tls" (the secure connection could not be verified),
        "not-found" or "checksum".

CookbookError(message: 'str', *, findings: 'list | None' = None)

The cookbook cannot be written because the project is not in sync.

Attributes:
    findings: The `check` findings that must be resolved first.

UsefulViewsWarning

Something worth knowing that does not stop the operation.


## Rules

The numbered rules a recipe must follow.


RULES

The numbered rules a recipe must follow: RULES[7] is rule 7, a Rule.

`RecipeRefused.rule` is a key of this dict, `useful-views rules` prints the
rules, and `describe_rules()` gives them as text.

Rule(number: ForwardRef('int'), title: ForwardRef('str'), detail: ForwardRef('str'))

One rule a recipe must follow.

Attributes:
    number: The rule's number, as in `RecipeRefused.rule`.
    title: A one-line title, as shown in refusal messages.
    detail: What the rule requires, in full.


----------------------------------------------------------------------
This is the User Guide documentation for the package.
----------------------------------------------------------------------


## Get started

### Installation

useful-views needs Python 3.11 or newer. Install it into the same Python
environment as the libraries your recipe steps use, such as polars or pandas:
steps run with the Python that runs useful-views.

## With uv (recommended)

[uv](https://docs.astral.sh/uv/) fetches a suitable Python for you, so it is
the easiest start on a new computer. In your project folder:

```sh
uv init --bare             # once: creates a pyproject.toml for this folder
uv add useful-views polars
```

`uv add` records each library in `pyproject.toml` and `uv.lock`. Add your
steps' other libraries the same way, for example `uv add pandas`. Run
commands with `uv run useful-views ...`, or activate the environment first
(`source .venv/bin/activate`; on Windows `.venv\Scripts\activate`).

Without a `pyproject.toml`, use a plain environment instead:

```sh
uv venv
uv pip install useful-views polars
```

## With pip

Check your Python first: `python3 --version` (on Windows, `py --version`)
must say 3.11 or newer. If it does not, install a newer Python from
[python.org](https://www.python.org/downloads/), or use uv.

macOS and Linux:

```sh
python3 -m venv .venv
source .venv/bin/activate
pip install useful-views polars
```

Windows (PowerShell):

```powershell
py -m venv .venv
.venv\Scripts\Activate.ps1
pip install useful-views polars
```

If PowerShell says that running scripts is disabled, run
`Set-ExecutionPolicy -Scope CurrentUser RemoteSigned` once, or use the
Command Prompt, where the second line is `.venv\Scripts\activate.bat`.

## With conda

useful-views is not on conda-forge yet. Make a conda environment with your
steps' libraries, then install useful-views into it with pip:

```sh
conda create -n my-project python=3.12 polars pyarrow
conda activate my-project
pip install useful-views
```

## Not as a separate tool

Do not run useful-views with `uvx`, `uv tool install` or `pipx`. Those give
it an environment of its own, and the steps, which run with the same Python,
would not find your project's libraries. Install it into the project's
environment, as above.

## In a notebook

The notebook must run in the same environment. Install Jupyter into it
(`uv add jupyterlab` or `pip install jupyterlab`) and start it from your
project folder, or in VS Code choose the environment's `.venv` as the
notebook's kernel. If `import useful_views` fails in a notebook, the notebook
is running a different Python.

## Record the environment

Commit the list of libraries your steps use, next to the project, so a
co-author or a reviewer can rebuild the same environment:

| Installed with | Commit | A co-author runs |
|---|---|---|
| uv, in a project | `pyproject.toml` and `uv.lock` | `uv sync` |
| pip | `requirements.txt`, from `pip freeze > requirements.txt` | `pip install -r requirements.txt` |
| conda | `environment.yml`, from `conda env export --from-history > environment.yml`, then add useful-views by hand (see below) | `conda env create -f environment.yml` |

`--from-history` lists only what you installed with conda, not what pip
installed. Add useful-views (and any other pip-installed library) to the
file yourself, under `pip`:

```yaml
dependencies:
  - python=3.12
  - polars
  - pyarrow
  - pip
  - pip:
      - useful-views
```

## Check the installation

```sh
useful-views --version
```

If the `useful-views` command is not found, `python -m useful_views
--version` runs the same program with the Python you name.

## Next

The [quickstart](quickstart.md) builds a ready-made example project and adds
a recipe of your own.


### Quickstart

This walkthrough starts from a ready-made example project, runs its recipes,
and then adds a recipe of your own: once in Python, as you would in a
notebook, and once on the command line. Each takes a few minutes. Do either
one, or both; they make separate projects.

The example data is an extract of the Penn World Table, version 11.0
(Feenstra, Inklaar and Timmer 2015), for Brazil, Germany, Spain and the
United States. It is licensed under CC BY 4.0; see [References](#references).

## Setup

Work in a new, empty folder. Install useful-views together with polars,
which the example's steps use, into one Python environment. Use uv or pip,
whichever you prefer.

With uv, which also fetches a suitable Python for you:

<!-- quickstart-test: skip -->
```sh
uv init --bare                     # creates a pyproject.toml for this folder
uv add useful-views polars
source .venv/bin/activate          # Windows: .venv\Scripts\activate
```

With pip, after checking that `python3 --version` says 3.11 or newer:

<!-- quickstart-test: skip -->
```sh
python3 -m venv .venv              # Windows: py -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install useful-views polars
```

[Installation](installation.md) covers Windows, conda and notebooks in more
detail.

Steps run with the same Python as useful-views, so whatever library your
steps import must be installed in that environment too: `uv add <package>`
or `pip install <package>`. (With uv you can also skip activating and put
`uv run` in front of each command, as in `uv run useful-views check`.)

For the notebook walkthrough, the notebook must run in this environment
too: install Jupyter into it (`uv add jupyterlab` or `pip install
jupyterlab`) and start it from this folder, or in VS Code choose `.venv` as
the notebook's kernel. If `import useful_views` fails in a notebook, the
notebook is running a different Python.

## In Python (a notebook)

Get the example project. `example` downloads its raw data and recipes from
the [useful-views-data](https://github.com/jameshorine/useful-views-data)
repository, checks each file against the checksum this version of
useful-views expects, and sets the folder up the way `init` would:

```python
import useful_views as uvs

root = uvs.example("pwt", "pwt-notebook")
```

It created this:

```text
pwt-notebook/
├── README.md          # what the example is, and the data credit
├── AGENTS.md          # instructions for AI coding assistants
├── datapackage.json   # the record of what each recipe made; empty for now
├── .gitignore         # keeps raw data, work files and views out of git
├── raw/
│   └── pwt110_bra_deu_esp_usa.csv
├── recipes/
│   ├── pwt_panel/     # recipe.toml and three steps
│   └── gdp_per_capita/
├── work/              # scratch files, emptied before each run
└── useful-views/
    └── README.md      # the catalog of views; empty until a recipe runs
```

For your own projects, `uvs.init("my-project")` makes the same structure,
empty, and you place the raw files in `raw/` by hand.

Run the first recipe:

```python
uvs.run("pwt_panel", root=root)
```

`run` checked the recipe, emptied `work/pwt_panel/`, ran the three steps in
order (their output appears as they run), checked that
`useful-views/pwt-panel.parquet` was written, and recorded the run in
`datapackage.json` and the catalog, `useful-views/README.md`. Had a step
failed, nothing would have been recorded, and the error would show the
step's own error output.

`run` does the one recipe you name. `sync` brings the whole project in line
with its recipe folders, like `uv sync`: it runs every recipe that is new,
changed or missing its views, upstream first. Here only `gdp_per_capita` is
left:

```python
result = uvs.sync(root=root)
print(result.ran)            # ('gdp_per_capita',)
print(uvs.check(root=root))  # [] means no problems
```

`datapackage.json` now records both recipes and the files they read and
made. Open the catalog, `useful-views/README.md`: for each view it gives the
recipe's description, its inputs, the numbered step descriptions (the top
comments of the step files), and the columns with their types.

In a notebook saved inside the project, such as `pwt-notebook/notebooks/`,
the working directory is that folder. `uvs.root()` finds the project from
there, so you can read a view with whichever library you like:

```python
import os
import polars as pl

os.makedirs(root / "notebooks", exist_ok=True)
os.chdir(root / "notebooks")  # Jupyter does this for a notebook saved there
gdp = pl.read_parquet(uvs.root() / "useful-views" / "gdp-per-capita.parquet")
print(gdp.filter(pl.col("year") == 2023))
```

### Write your own recipe

Add a recipe that reads the `gdp-per-capita` view and averages each
country's growth since 2000. Start from the template:

```python
folder = uvs.new_recipe("growth_since_2000", root=root)
```

That writes `recipe.toml` and a stub step, `01_first_step.py`. Both still
say TODO, so running the recipe is refused before anything changes, with a
message that says what to fix and where:

```python
try:
    uvs.run("growth_since_2000", root=root)
except uvs.RecipeRefused as err:
    print(err)
```

<!-- quickstart-output: refusal -->
```text
recipes/growth_since_2000/recipe.toml needs fixing:
  - `description` still starts with TODO: replace it with one sentence saying what this recipe produces
  - `inputs` is empty: list at least one path, like inputs = ["raw/my_data.csv"]
  - `outputs` is empty: list at least one path, like outputs = ["useful-views/my-view.parquet"]
(Rule 1: recipe.toml is complete. To see every rule: useful-views rules)
```

Normally you would now edit the two files in your editor. Here Python writes
them, so the walkthrough runs as it stands. The step's first line is its
description, and its paths are the same strings as in `recipe.toml`:

```python
(folder / "recipe.toml").write_text(
    'description = "Average annual growth of GDP per capita since 2000."\n'
    'inputs  = ["useful-views/gdp-per-capita.parquet"]\n'
    'outputs = ["useful-views/growth-since-2000.parquet"]\n'
)
(folder / "01_first_step.py").unlink()
(folder / "01_average_growth.py").write_text(
    "# Average each country's annual growth of GDP per capita from 2000 on.\n"
    "import polars as pl\n"
    "\n"
    'gdp = pl.read_parquet("useful-views/gdp-per-capita.parquet")\n'
    "growth = (\n"
    '    gdp.filter(pl.col("year") >= 2000)\n'
    '    .group_by("iso", "country")\n'
    '    .agg(pl.col("growth_pct").mean().round(2).alias("avg_growth_pct"))\n'
    '    .sort("iso")\n'
    ")\n"
    'growth.write_parquet("useful-views/growth-since-2000.parquet")\n'
)
print(uvs.sync(root=root).ran)  # ('growth_since_2000',)
growth = uvs.root() / "useful-views" / "growth-since-2000.parquet"
print(pl.read_parquet(growth))
```

The catalog now lists three views.

## On the command line

The same, from a terminal:

```sh
useful-views example pwt pwt-cli
cd pwt-cli
useful-views run pwt_panel
useful-views sync
useful-views check
useful-views cookbook
cat useful-views/README.md
```

`sync` runs only `gdp_per_capita`, since `pwt_panel` is already up to date,
and `useful-views check` prints `No problems found.` `useful-views cookbook`
writes `cookbook.json`, the list of every recipe and its steps in run order
(see [Cookbook](concepts/cookbook.md)).
To add a recipe of your own, `useful-views new-recipe <name>` writes the
template; edit its two files as above, then `useful-views sync`.

Steps are code: `run` and `sync` execute the Python in `recipes/` with your
permissions. Read the recipes of a project you did not write before running
them.

## Start your own project

For a real project, keep the environment files in the project folder, so
they are committed with it:

<!-- quickstart-test: skip -->
```sh
mkdir my-project
cd my-project
uv init --bare
uv add useful-views polars
uv run useful-views init
```

`init` makes the structure next to `pyproject.toml`. Put your raw files in
`raw/`, say in `README.md` where each came from, and add recipes with
`useful-views new-recipe <name>`.

## What to commit

`git init` in the project folder, then commit everything git does not
ignore: `README.md`, `AGENTS.md`, `recipes/`, `datapackage.json`,
`cookbook.json`, `.gitignore`, `raw/.gitkeep` and `useful-views/README.md`,
and the files that list your steps' libraries (`pyproject.toml` and
`uv.lock`, or `requirements.txt`). The raw data, `work/` and the parquet
views stay out of git. Before each commit, run `useful-views sync`, then
`useful-views cookbook`.

A teammate who clones the project installs the libraries (`uv sync`),
places the raw files as your `README.md` describes, then runs
`useful-views sync`, which makes every view again, upstream first.

## Changing things later

| When you... | Do this |
|---|---|
| add a recipe, or edit a `recipe.toml` or a step's top comment | `useful-views sync`: it runs that recipe and everything downstream of it |
| edit the code inside a step | `useful-views sync --all`, or `run` that recipe and then each recipe downstream of it. (`sync` alone does not notice code changes yet.) |
| replace or correct a file in `raw/` under the same name | `useful-views sync --all` (or give the new version a new file name and update `inputs`, which `sync` notices) |
| `git pull`, merge, or switch branches | `useful-views sync --all`: the views are not in git, so they still come from the recipes as they were |
| delete a recipe folder, or remove an output from a recipe | `useful-views sync`: it removes the old records and view files (first change the inputs of any recipe that reads those views) |
| rename a recipe folder | change `work/<old name>/` to `work/<new name>/` in its steps, then `useful-views sync` |
| clone the project on another computer | install the libraries, place the raw files, then `useful-views sync` |
| want to know if anything is missing or broken | `useful-views check`. It notices edited `recipe.toml` files, added or removed steps and changed top comments, but not edits to a step's code or to a file's contents. |

## When something is refused

Every refusal names the file to fix and the rule it broke, and says how to
fix it. `run` and `sync` both refuse before anything on disk changes: `sync`
checks every recipe first and lists every problem at once. Only a failure
while a recipe is running (a step that fails, or a view that was not
written) stops a `sync` part way; what it had already done is kept, and the
message says what that was and which command to run next. The common ones:

- **An input does not exist**: check the path in `recipe.toml` against the
  file, including capital letters. If it is another recipe's view, run that
  recipe first, or use `sync`.
- **A path is written in the wrong form**: paths are relative to the project
  root with forward slashes, like `raw/data.csv`. The message gives the
  corrected string where it can.
- **A step has no top comment**: start the step file with a `#` comment
  saying what it does. A docstring does not count.
- **A step failed**: this is not a refusal. The run had started, so the
  recipe's views were removed and nothing was recorded; `work/<recipe>/`
  shows how far it got. The message shows the last lines of the step's error
  output. A `ModuleNotFoundError` means the package is not installed in the
  environment useful-views runs in; the message gives the install command.

[Rules](reference/rules.md) lists every rule; so does `useful-views rules`.

## References

Feenstra, Robert C., Robert Inklaar and Marcel P. Timmer (2015), "The Next
Generation of the Penn World Table." *American Economic Review*, 105(10),
3150–3182. https://doi.org/10.1257/aer.20130954

The example uses Penn World Table version 11.0, available at
https://www.rug.nl/ggdc/productivity/pwt/ under a Creative Commons
Attribution 4.0 International License. The extract was exported on
2026-10-09 from the PWT online data access tool; the example project's
`README.md` gives the exact source. Cite the paper above whenever you use
PWT data.



## Concepts

### Projects

A useful-views project is a folder with a fixed structure. Every project
looks the same, so a teammate, a reviewer or an AI tool knows where
everything is.

```text
my-data-project/
├── README.md          # written by you: the project and its data sources
├── AGENTS.md          # instructions for AI coding assistants; written once by init
├── datapackage.json   # generated: what each recipe made from what
├── cookbook.json      # generated by useful-views cookbook: the recipes used
├── .gitignore         # written once by init
├── raw/               # raw inputs, placed by hand; never written by a recipe
├── recipes/           # one folder per recipe, written by you
├── work/              # in-between files, one folder per recipe
└── useful-views/      # the declared views, plus the generated catalog
```

Create one with `useful-views init my-data-project`, or start from a
ready-made example with `useful-views example pwt`.

## What lives where

| Kind | Where | Made by | In git |
|---|---|---|---|
| Raw input | `raw/` (subfolders allowed) | You, by hand | No |
| Work file | `work/<recipe>/` | A recipe's steps, for its own later steps | No |
| Useful view | `useful-views/`, flat | A recipe, as a declared output | No (parquet); the catalog yes |
| Record | `datapackage.json`, `useful-views/README.md` | useful-views, after each successful run | Yes |
| Cookbook | `cookbook.json` | `useful-views cookbook`, from the record | Yes |

- **Raw inputs** are the data you start from. Nothing a recipe does writes to
  `raw/`. Because `raw/` is not committed, say in the project's `README.md`
  where each file came from.
- **Work files** pass data from one step to the next within a recipe. No
  other recipe may read them, and `run` empties `work/<recipe>/` before each
  run, so a leftover never passes for a fresh result.
- **Useful views** are the derived tables you actually use. Each is a
  `.parquet` file made by exactly one recipe, and another recipe may read it
  as an input.
- **Records** say which recipe made each view, from which inputs, with which
  steps. See [Records](records.md).

## Who writes what

You write the project `README.md` and everything in `recipes/`; `init` and
`new-recipe` only start them from templates and never touch them again.
useful-views writes `datapackage.json`, `useful-views/README.md` and
`cookbook.json`; never edit those by hand. `init` writes `.gitignore` and
`AGENTS.md` once, if they are not there; after that they are yours.

## What to commit

Commit `README.md`, `AGENTS.md`, `recipes/`, `datapackage.json`,
`cookbook.json`, `.gitignore`, `raw/.gitkeep` and `useful-views/README.md`,
and the files that list your steps' libraries (`pyproject.toml` and
`uv.lock`, `requirements.txt` or `environment.yml`; see
[Installation](../installation.md#record-the-environment)). The generated
`.gitignore` keeps the rest out:

```text
raw/*
!raw/.gitkeep
/work/
useful-views/*.parquet
__pycache__/
```

If the folder already had a `.gitignore`, `init` leaves it alone and warns
if git would not ignore raw data, work files or views, naming the lines to
add.

A teammate who clones the project installs the libraries, places the raw
files as your `README.md` describes, then runs `useful-views sync`, which
makes every view again.

## Sharing raw data

`raw/` is not in git, because raw data is often large, private or licensed.
Share it the way its license allows: the original source (with the exact
download date, in your `README.md`), your institution's storage, or a data
repository. The record in `datapackage.json` lists every raw file each
recipe read, with its columns, so a teammate can check they have the right
files.

## Finding the project from a subfolder

Every command and function finds the project by walking up from the current
folder to `datapackage.json`, so they work from a notebook in `notebooks/`.
From anywhere else, name it: `useful-views -C path/to/project check`, or
`root=` in Python.

`init` refuses to make a project in your home folder or at the top of a
drive, since every folder below would then be inside it (`--force` if you
really mean it). Projects cannot be nested.


### Steps

A **step** is one Python file in a recipe folder, and the smallest unit of
work in a project. A step does at least one real transformation of the
data. There is no maximum: it may do many, and you decide where one step
ends and the next begins. A [recipe](recipes.md) runs its steps in order.

```python
# Reshape the wide export (one column per year) into one row per country,
# variable and year.
import polars as pl

wide = pl.read_csv("raw/pwt110_bra_deu_esp_usa.csv")
long = wide.unpivot(
    index=["ISO code", "Country", "Variable code", "Variable name"],
    variable_name="year",
    value_name="value",
).with_columns(pl.col("year").cast(pl.Int32))
long.write_parquet("work/pwt_panel/long.parquet")
```

A step is an ordinary script. It does not import `useful_views`, and it can
use any data library installed in the environment.

## Names

A step file is named `NN_name.py`: two digits, an underscore and a name, such
as `01_reshape_long.py`. Steps run in numeric order; `00_` and gaps in the
numbering are fine, so you can add a step between `01_` and `02_` later by
renumbering, or leave room (`10_`, `20_`, ...). Two steps may not share a
number.

A file named almost like a step is not run, and `run` warns about it:
`1_clean.py`, `001_clean.py`, `01-clean.py`, `01 clean.py`, `01_clean.PY`,
`01_clean` with no extension, or `01_clean.py.txt` (an extension your editor
added). An R, notebook, Quarto, Julia or Stata file named like a step
(`02_clean.R`, `02_explore.ipynb`, ...) is refused: steps must be Python
files. Any other file in the recipe folder, such as notes, editor backups
(`.bak`, `~`, `.swp`) or a helper module like `helpers.py`, is ignored, with
two exceptions. A `.py` file whose name starts with a digit, such as
`2020_codes.py`, is not run but gets the warning above. A copy that keeps
the `NN_` prefix and `.py`, such as `01_clean copy.py` or `09_old_clean.py`,
is a step: it is refused if its number is taken, and run otherwise. Keep old
versions of a step outside the recipe folder.

## The description

The block of `#` lines at the very top of the file is the step's
description. Its lines are joined with spaces into one line, so a sentence
can wrap across several `#` lines. It is required, and must not start with
TODO. It becomes the step's description in `datapackage.json`, in the catalog
and in `cookbook.json`, so write it for a reader who has not seen the code:
what the step does to the data, not how.

Lines before it are skipped: blank lines, a shebang (`#!`), a coding line,
notebook cell markers (`# %%`, `# In[1]:`), a Jupytext header, an inline
script metadata block (`# /// script` ... `# ///`) and decoration lines such
as `# -----`. A cell marker is skipped as a whole line, including any title
written after `# %%`, so put the description on its own `#` lines below the
marker. A blank line, a cell marker or a decoration line after the
description ends it.

Any other `#` line at the top counts as description, including a licence
header (`# SPDX-License-Identifier: MIT`) and linter pragmas (`# ruff:
noqa`): put the description first, then a blank line, then those. A
docstring is not skipped and is never read as the description: put the `#`
comment before any docstring or code, or the step is refused as having no
top comment.

A step file must be saved as UTF-8 (a byte-order mark and Windows line
endings are fine). A file in another encoding is refused, even with a coding
line that declares it.

Changing a step's description is a change `sync` notices: it runs the recipe
again, and everything downstream of it.

## How a step runs

`run` and `sync` run each step as its own process:

- from the project root, so paths in the code are the same strings as in
  `recipe.toml`: `raw/...`, `work/<recipe>/...`, `useful-views/...`;
- with the same Python as useful-views, so the libraries it imports must be
  installed in that environment;
- with `work/<recipe>/` created and empty before the first step;
- with its output shown as it runs, in a terminal or a notebook cell;
- in UTF-8 mode, so `open()` reads and writes UTF-8 by default on every
  system (pass `encoding=` for a file in another encoding);
- with no keyboard input, so `input()` and `breakpoint()` fail.

The first step that fails stops the run. See
[Environment](../reference/environment.md) for every detail.

## What a step may read and write

A step reads its recipe's inputs and the files earlier steps of the same
recipe wrote to `work/<recipe>/`. It writes in-between files to
`work/<recipe>/`, in any format, and the recipe's views to the outputs
declared in `recipe.toml`.

It never writes to `raw/`, never writes another recipe's views, and never
reads another recipe's work files: to use another recipe's result, declare
its view as an input of yours. useful-views does not stop a step from doing
otherwise. While a recipe runs, it watches the files in `raw/` that recipes
declare or have recorded as inputs, and the other recipes' views, and warns
if a step changed or deleted one. It does not notice a new file in `raw/`, or
a step reading another recipe's work files. `check` reports a file in
`useful-views/` that no recipe declares as a stray file.

## Steps are code

`run` and `sync` execute steps as you: with your permissions, your
environment variables (including any passwords or tokens in them) and your
network access, with no sandbox. Read the steps of a project you did not
write before running them. See [run, sync and check](running.md).

## Code shared between steps

A step can import a module that sits next to it in its recipe folder, such
as `recipes/pwt_panel/helpers.py`; useful-views ignores files that are not
named like a step. The project root is not on a step's import path, so a
step cannot import a module that sits there. (Steps do inherit your
`PYTHONPATH`, so setting it can make such an import work for you, but the
project then fails for anyone without that setting.) Code several recipes
share belongs in a small package installed into the environment.

Editing a helper module is like editing a step's code: `sync` does not
notice it, so use `useful-views sync --all`.

## When a step fails

Nothing is recorded, the recipe's views are removed, and its record from the
last successful run is kept. The files in `work/<recipe>/` stay as the steps
left them, so you can see how far it got. The message shows the last lines
of the step's error output and the command to run again. Two failures get
their own advice:

- `ModuleNotFoundError`: the message names the Python the step ran with and
  how to install the package into it, or says that the module is your own
  code in the wrong place.
- A step stopped by the system (on Linux and macOS, `SIGKILL`): most often
  it ran out of memory. Process the data in smaller pieces, or ask for more
  memory.

## Running a step by hand

To debug a step, run it yourself from the project root, with the same Python
as useful-views, as `run` does:

```sh
python -X utf8 recipes/pwt_panel/01_reshape_long.py
```

Use the Python of the environment useful-views is installed in: activate it,
or use `uv run python ...`. A step failure from `run` names that interpreter.
`-X utf8` turns on the UTF-8 mode `run` uses. A step also expects what `run`
prepares: `work/<recipe>/` must exist (`run` creates it; before the recipe's
first run, create it yourself), and a later step reads the files earlier
steps left there, so run the recipe, or the earlier steps, first.

Many editors run a file from its own folder by default, so set the working
directory to the project root: in VS Code, open the project folder and, for
the Interactive window, set `"jupyter.notebookFileRoot":
"${workspaceFolder}"`; in PyCharm, set the run configuration's working
directory; in Spyder, choose the current working directory under Run
settings. An editor console started at the project root has the project
root on its import path instead of the step's recipe folder, so a helper
module next to the step does not import there; run the file as a script
instead.

A step run by hand writes its files but records nothing, and `sync` and
`check` do not notice it: they report everything in sync while
`datapackage.json` and the catalog still describe the views from the last
recorded run. Afterwards, run `useful-views run <recipe>` and then each
recipe downstream of it (`run` names them), or `useful-views sync --all`.

Rules 2 and 3 are about steps; [Rules](../reference/rules.md) lists them all.


### Recipes

A **recipe** is a folder in `recipes/`: a `recipe.toml` and one or more
[steps](steps.md). From the inputs it declares, it makes the views it
declares, by running its steps in order. Every view is made by exactly one
recipe, and records that recipe.

```text
recipes/pwt_panel/
├── recipe.toml                 # description, inputs, outputs
├── 01_reshape_long.py          # the steps, run in numeric order
├── 02_keep_core_variables.py
└── 03_pivot_to_panel.py
```

## Making one

```sh
useful-views new-recipe pwt_panel
```

This writes `recipes/pwt_panel/recipe.toml` and a stub step,
`01_first_step.py`. Both say TODO, and the recipe is refused until you
replace them. A recipe's name is its folder's name: lowercase letters,
digits and underscores, such as `pwt_panel`. It may not be a name Windows
reserves for devices (`con`, `aux`, `nul`, `prn`, `com1`, ...), or
`__pycache__`, which useful-views ignores.

## recipe.toml

```toml
description = "Country-year panel of core Penn World Table variables."
inputs  = ["raw/pwt110_bra_deu_esp_usa.csv"]
outputs = ["useful-views/pwt-panel.parquet"]
```

- `description`: one sentence saying what the recipe produces. It is the
  first thing the catalog and the cookbook say about the recipe's views.
- `inputs`: at least one file the steps read: a file under `raw/`, or a view
  another recipe makes.
- `outputs`: at least one `.parquet` file directly inside `useful-views/`:
  the views the steps write.

No path may be listed twice. Other keys are ignored with a warning, so check
the spelling.

## Inputs and outputs

A raw input is a file you place in `raw/` by hand; it may sit in a subfolder
and be in any format that is a single file. A view input is another recipe's
output. Reading it makes that recipe **upstream** of this one: `sync` runs
it first, and runs this one again whenever it runs the upstream one again.
`useful-views run` on the upstream recipe does not: it warns and names this
one, and a later `sync` or `check` does not notice, so use `sync --all` or
run this one too. Recipes may not form a cycle, and a recipe may not read
its own view.

An output is a view that only this recipe makes. Views are parquet files, so
the record can say exactly which columns and types each one has. In-between
files, in any format, belong in `work/<recipe>/` instead; see
[Steps](steps.md).

Every raw input and every view also gets a resource name: a view's file
name, or a raw file's path below `raw/`, without the extension and
lowercased (`raw/wave1/survey.csv` is `wave1-survey`; see
[Records](records.md)). No two files recorded anywhere in the project may
share one ([rule 8](../reference/rules.md)). So a recipe cannot read
`raw/gdp.csv` and write `useful-views/gdp.parquet` (call the view
`gdp-panel.parquet`), and cannot read both `raw/survey.csv` and
`raw/survey.dta`.

A recipe needs at least one input. To use data from a database or a web API,
save an extract into `raw/` first, with a script outside `recipes/`, note in
the project's `README.md` when and how, and read that file.

## Paths

Paths are relative to the project root, use forward slashes, and match the
letter case on disk: `raw/data.csv`, not `./raw/data.csv`, `raw\data.csv` or
an absolute path. A path in `recipe.toml` is the same string the steps use
in their code, and the same string another recipe uses to read the view.
When a path is written wrongly, the refusal gives the corrected string where
it can.

Each path names one file. There are no wildcards, so list every file a
recipe reads (`raw/2024-01.csv`, `raw/2024-02.csv`, ...). Formats made of a
folder (a partitioned parquet dataset, a Zarr store) or of several files read
together (a shapefile's `.shp`, `.dbf` and `.shx`, which share one resource
name) are not supported yet.

Every folder and file name in a path must be one that can exist on Windows:
no `<`, `>`, `:`, `"`, `|`, `?` or `*`, not ending in a dot or a space, and
not a device name such as `con`, `aux` or `nul`, with or without an
extension. Rename such files before listing them. Spaces inside a name are
fine.

## Changing a recipe

`sync` compares each recipe folder with its record from the last successful
run. It notices a changed description, inputs or outputs in `recipe.toml`, a
step added, removed or renamed, and a step's changed description, and runs
the recipe again, with everything downstream of it. An output you only
remove is different: `sync` deletes that view and drops it from the record,
without running the recipe again.

It does not notice edits inside a step's code, a helper module, or a raw
file replaced under the same name. Nor does it notice a recipe change that
arrives together with its record from a `git pull`, merge or branch switch:
the record then already matches the folder, while the views, which are not
in git, still come from the recipe as it was. After any of those, use
`useful-views sync --all`.

To rename a recipe, rename its folder, change `work/<old name>/` to
`work/<new name>/` in its steps, then run `useful-views sync`. To delete
one, delete its folder and run `useful-views sync`, which removes its record,
its views and its work files. If another recipe reads one of those views,
change that recipe's `inputs` first, or delete it too: otherwise its own
views stay as they were, and only `check` reports the missing input. The
same goes for removing an output that another recipe reads. See
[run, sync and check](running.md).

Rules 1 and 4 to 8 are about recipes; [Rules](../reference/rules.md) lists
them all.


### run, sync and check

`run` and `sync` differ the way `uv run` and `uv sync` do: `run` does the one
recipe you name, now; `sync` brings the whole project in line with its
recipe folders. `check` reports, and changes nothing.

| You want to | Use |
|---|---|
| Run one recipe now (what it reads must already be made) | `useful-views run <recipe>` |
| Make everything that is missing or changed, in order | `useful-views sync` |
| Rebuild every view from scratch, in order | `useful-views sync --all` |
| Know what is wrong, or what sync would do | `useful-views check` |

## Steps are code

`run` and `sync` execute every step in `recipes/` as an ordinary Python
program, as you: with your permissions, your environment variables
(including any passwords or tokens in them) and your network access. There
is no sandbox. Read the recipes of a project you did not write, or a change
someone proposes, before you `run` or `sync` it, as you would read any code
before running it. `check`, `rules` and `cookbook --check` only read files,
and `cookbook` writes only `cookbook.json`; none of them runs a step.

Steps must never write to `raw/`, or to another recipe's views or work
files. useful-views does not stop a step from doing so; the rules and the
record only work if steps keep to their own outputs.

## run

Before changing anything, `run` checks the recipe against the
[rules](../reference/rules.md). Then it:

1. deletes the recipe's declared views and empties `work/<recipe>/`, so a
   leftover file can never pass for a fresh result;
2. runs each step in order, showing its output as it runs, in a terminal or
   a notebook cell. The first step that fails stops the run;
3. checks that every declared view was written as a non-empty parquet file;
4. records the run in `datapackage.json` and the catalog. Only this recipe's
   entries change; every other recipe's record stays exactly as it was.

If a step fails or a view is missing, nothing is recorded, the recipe's views
are removed, and its record from the last successful run, if any, is kept.
The files in `work/<recipe>/` stay as the steps left them. Running a recipe
twice leaves `datapackage.json` and the catalog byte-for-byte unchanged.

`run` never runs other recipes. If an input is another recipe's view, that
recipe must have run first; `sync` handles the order for you. If other
recipes read the views `run` just made again, it warns and names them: their
views were made from the old ones, and `sync` does not notice that by itself.
Run `useful-views sync --all`, or run each of them, upstream first.

## sync

`sync` makes the project match its recipe folders, the way `uv sync` makes
an environment match its lock file. Records only ever come from real runs.

1. It checks every recipe against the rules, judging each against the
   records as they will be when its turn comes. If any recipe would be
   refused, it lists every problem and changes nothing.
2. It removes the records of recipes whose folder was deleted, and any view
   a recipe no longer lists, along with those view files.
3. It runs, upstream first, every recipe that has never run, whose
   `recipe.toml` or steps no longer match its record, or whose views are
   missing, and everything downstream of those.

If a recipe fails, or sync is interrupted, it stops there. What ran before
stays recorded, and the message says what sync had done, what it had not
reached, and which command to run next. Recipes it had not reached that read
views it had already made again would be left out of date, so their views
are removed: the next `sync` sees them as missing and makes them again.

On a fresh clone, place the raw files and run `useful-views sync`.

`sync` does not notice a change inside a step's code, or a raw file replaced
under the same name, since that needs hashing, which is planned for a
future release. After such a change, use `useful-views sync --all` (or
`run` that recipe and then each recipe downstream of it). The same goes
after a `git pull`, merge or branch switch: the views are not in git, so
they still come from the recipes as they were before.

## One command at a time

`run`, `sync` and `cookbook` (when it writes) hold a lock on the project
while they work, in `work/.useful-views.lock`, so two of them started at
once cannot overwrite each other's records. A second one is refused with a
message saying another command is changing the project; wait, and run it
again. The operating system releases the lock when the command ends, even if
it crashes. `check`, `rules` and `cookbook --check` do not take it. The lock
file is the one thing a refused `run` or `sync` leaves behind (in `work/`,
which is not in git).

## check

`check` changes nothing. It reports what is missing or out of place, every
rule a recipe breaks, and every recipe `sync` would run, each with what to do
about it. See [Check findings](../reference/check-findings.md). It exits with
1 when it finds anything, so it works as a gate in scripts, and in CI jobs
that have the raw data and run `useful-views sync` first. A plain git
checkout has no raw files or views, so there `check` always reports them
missing; to gate such a job, use `useful-views cookbook --check` (see
[Cookbook](cookbook.md#checking-it-in-ci)).

## When you change things

| When you... | Do this |
|---|---|
| add a recipe, or edit a `recipe.toml` or a step's top comment | `useful-views sync` |
| edit the code inside a step | `useful-views sync --all` |
| replace or correct a file in `raw/` under the same name | `useful-views sync --all` (or give the new version a new file name and update `inputs`; then `sync` alone notices) |
| `git pull`, merge, or switch branches | `useful-views sync --all` |
| delete a recipe folder, or remove an output | `useful-views sync`: it removes the old records and view files (first change the inputs of any recipe that reads those views) |
| rename a recipe folder | change `work/<old name>/` to `work/<new name>/` in its steps, then `useful-views sync` |
| clone the project on another computer | place the raw files, then `useful-views sync` |
| finish a change you will commit | `useful-views sync`, then `useful-views cookbook`, then commit |


### Records

After each successful run, useful-views rewrites two files: the record,
`datapackage.json`, and a readable catalog, `useful-views/README.md`. Never
edit them by hand; if one is damaged, restore it from git.

If a merge leaves git conflict markers in them, do not resolve the conflict
by hand: take either side (`git checkout --ours datapackage.json
useful-views/README.md`, and `cookbook.json` if the project has one), then
run `useful-views sync --all` (and `useful-views cookbook`), and `git add`
the files. useful-views says so when it finds markers in `datapackage.json`;
markers in `cookbook.json` alone are reported as `stale-cookbook`.

## datapackage.json

The record is a [Data Package](https://datapackage.org/), a standard format
many tools can read. `frictionless validate datapackage.json` passes on it
whenever every recorded file is present. It sits at the project root because
it describes files in both `raw/` and `useful-views/`.

- `resources` lists every raw file a recorded recipe read, then every view,
  each with its columns and their types. A view also has `producedBy`: the
  recipe that made it and that recipe's inputs.
- `recipes` lists each recorded recipe with its description, inputs,
  outputs and steps, as they were at its last successful run.

```json
{
  "name": "gdp-per-capita",
  "type": "table",
  "path": "useful-views/gdp-per-capita.parquet",
  "scheme": "file",
  "format": "parquet",
  "mediatype": "application/parquet",
  "schema": {
    "fields": [
      {"name": "iso", "type": "string"},
      {"name": "year", "type": "integer"}
    ]
  },
  "producedBy": {
    "recipe": "gdp_per_capita",
    "inputs": ["useful-views/pwt-panel.parquet"]
  }
}
```

Column types come from the files themselves: a parquet file's own types, and
for a raw CSV, types inferred from the whole file. In a raw CSV only an empty
cell counts as missing, so a column that marks missing values with `NA` or
`.` is recorded as `string`. Other raw formats (`.dta`, `.xlsx`, ...), and
CSVs that are not plain comma-separated UTF-8, are recorded without columns.

A resource's name is a view's file name, or a raw file's path below `raw/`,
without the extension, lowercased, with `/` and other characters than
`a-z`, `0-9`, `.`, `_` and `-` turned into `-`: `raw/wave1/survey.csv` is
`wave1-survey`. Accented letters lose their accents, and letters of other
scripts are written as their Unicode code point, so names stay distinct. Two
recorded files may not share one ([rule 8](../reference/rules.md)).

## useful-views/README.md

The catalog is generated from `datapackage.json` alone. For each view it
gives the recipe's description, its inputs, its numbered steps (their top
comments) and its columns with their types. It is committed to git, so a
reviewer sees what each view is and how it was made without running
anything.

## cookbook.json

From this record, `useful-views cookbook` writes a third file you commit:
`cookbook.json`, the ledger of every recipe the project uses, in run order,
with its steps. See [Cookbook](cookbook.md).

## What the record does not track yet

The record says which recipe made each view, from which inputs, with which
steps. It does not yet record checksums of the files, so it cannot tell that
a raw file, a step's code or a view changed after the run. That is planned
for a future release.


### Cookbook

A **cookbook** is a collection of recipes. In a useful-views project it is
`cookbook.json` at the project root: the ledger of every recipe the project
uses, in the order they run, with each recipe's steps. It is written by a
command, the way `uv lock` writes `uv.lock`, and never edited by hand.

```sh
useful-views cookbook           # write or update cookbook.json
useful-views cookbook --check   # exit 1 if it is missing or out of date
```

From Python: `uvs.cookbook()` and `uvs.cookbook(check=True)`.

## Why keep one

The record in `datapackage.json` is written for tools: every file, its
columns and types, recipes in name order. The cookbook is written for
people and reviewers: one file that reads top to bottom as the method of the
project, upstream first, and that changes in git when a recipe's
`recipe.toml`, its list of steps or a step's top comment changes (not when
only the code inside a step changes; see [What is next](#what-is-next)). A reviewer reading a pull request, an examiner reading a thesis
repository, or an AI assistant opening the project sees what is made from
what without running anything.

| To find out | Read |
|---|---|
| the method: which recipes, in which order, with which steps | `cookbook.json` |
| what one view contains, its columns and their types | `useful-views/README.md` (the catalog) |
| everything, in a standard format for tools | `datapackage.json` |

## When to write it

After the views are made, and before you commit:

```sh
useful-views sync
useful-views cookbook
git add cookbook.json datapackage.json useful-views/README.md recipes/
```

`cookbook` writes only when the project is in sync, since the cookbook
describes recipes as they last ran. If `check` reports anything other than
stray files or a stale catalog or cookbook, `cookbook` refuses, lists the
problems, and writes nothing. Writing an unchanged cookbook leaves the file
and its timestamp alone, so git shows a change only when there is one.

`run` and `sync` do not update the cookbook themselves. Once the project has
a `cookbook.json`, they warn when they change the records and leave it out of
date, and `check` reports `stale-cookbook` until you run `useful-views
cookbook`. Before the first `useful-views cookbook`, nothing reminds you:
`check` says `No problems found.`, but `cookbook --check` (and so the CI job
below) exits with 1 until `cookbook.json` is written and committed.

## Checking it in CI

`cookbook --check` needs no data: the raw files and the views are not in
git, so a CI job only has the recipes and the generated files. It exits
with 1, and lists why, when:

- `cookbook.json` is missing, or differs from the committed
  `datapackage.json`;
- a recipe folder no longer matches its record: a recipe added or removed,
  its `recipe.toml` edited, a step added, removed or renamed, a step's top
  comment changed, or an output dropped, and committed without running
  `sync`;
- a recipe breaks a rule (`refused`), or a folder in `recipes/` has no
  readable `recipe.toml` (`unreadable-recipe`). `sync` does not fix these:
  fix what is listed first.

An edit inside a step's code passes, as it passes `check` and `sync` (see
[What is next](#what-is-next)). The catalog, `useful-views/README.md`, is
not checked; `useful-views sync` writes it again. If `datapackage.json`
cannot be read, for example because of merge conflict markers, the command
exits with 3.

A GitHub Actions job that requires an up-to-date cookbook:

```yaml
name: Cookbook
on: [push, pull_request]
permissions:
  contents: read
jobs:
  cookbook:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - uses: actions/setup-python@v6
        with:
          python-version: "3.12"
      - run: pip install useful-views==0.1.0   # the version your project uses
      - run: useful-views cookbook --check
```

The job assumes the project is at the top of the repository. If it is in a
subfolder, name it: `useful-views -C path/to/project cookbook --check`.
Without that, the command finds no project and exits with 3, not 1.

It runs no step, so the job needs neither your data nor your steps'
libraries. Pin useful-views to the version in your environment file:
`--check` compares the committed file with what the installed version would
write, so an unpinned job could fail on an unchanged project when a new
release changes the format.

With `--json`, `ok` is the answer. `findings` lists the recipe folders that
no longer match their records, as the same objects `check` gives; it is
empty when the only problem is that `cookbook.json` is missing or differs,
so test `ok`, not `findings`. When `useful-views cookbook` (without
`--check`) refuses to write, it exits with 3, and its findings are in
`error.findings`.

## What is in it

```json
{
  "version": 1,
  "generator": "useful-views",
  "project": "pwt-example",
  "raw_inputs": ["raw/pwt110_bra_deu_esp_usa.csv"],
  "recipes": [
    {
      "name": "pwt_panel",
      "description": "Country-year panel of core Penn World Table variables for ...",
      "path": "recipes/pwt_panel",
      "inputs": ["raw/pwt110_bra_deu_esp_usa.csv"],
      "outputs": ["useful-views/pwt-panel.parquet"],
      "steps": [
        {
          "name": "reshape_long",
          "path": "recipes/pwt_panel/01_reshape_long.py",
          "description": "Reshape the wide export (one column per year) into ..."
        }
      ]
    }
  ]
}
```

(Shortened: the real file lists every step, and `gdp_per_capita` after
`pwt_panel`, one list item per line.)

| Field | What it holds |
|---|---|
| `version` | The format's version: 1 |
| `generator` | `useful-views` |
| `project` | The package name recorded in `datapackage.json`. It is chosen once, from the folder's name when the project is created, and kept, so a clone in a folder with another name writes the same file. |
| `raw_inputs` | Every file under `raw/` a recorded recipe reads, sorted |
| `recipes` | Every recorded recipe, upstream first, each exactly as its record in `datapackage.json`: name, description, path, inputs, outputs and steps |

## Common situations

- **`check` reports `stale-cookbook`**: the records changed since the
  cookbook was written. Run `useful-views sync`, then `useful-views
  cookbook`, and commit both.
- **A merge conflict in `cookbook.json`**: do not resolve it by hand;
  `datapackage.json` and `useful-views/README.md` usually conflict too. Take
  either side of each generated file (`git checkout --ours datapackage.json
  useful-views/README.md cookbook.json`), run `useful-views sync --all` (it
  runs every recipe again, so it needs the raw data), then `useful-views
  cookbook`, and `git add` those files. A cookbook that is not valid JSON, or
  not UTF-8, counts as out of date, and `cookbook` writes it again. See
  [Records](records.md).
- **A co-author's clone shows a changed cookbook**: it should not, if both of
  you ran `sync` and `cookbook` on the same recipes with the same version of
  useful-views. A difference means the recipes differ, one of you committed
  without running `cookbook`, or you use different versions of useful-views.
  `git diff cookbook.json` shows which recipe and step changed. Keep everyone
  on the version pinned in the project's environment file.

## What is next

Version 1 says which recipes and steps made the views. It does not yet
record checksums, so it cannot tell that a raw file, a step's code or a view
changed since. Later versions are planned to add the SHA-256 of every raw
input, recipe file, step and view, and then the environment the steps ran
in, so that `cookbook --check` can tell when anything changed, the way
`uv.lock` locks a Python environment.



## Guides

### Working with AI tools

useful-views is built to be driven by an AI coding assistant as well as by a
person: plain files, rules checked before anything changes, messages that
name the file to fix, and machine-readable output. This page is for the
assistant, or for you when you instruct one.

## Instructions in every project

`useful-views init` writes an `AGENTS.md` into each new project: a short set
of instructions that many coding assistants read by themselves when they open
a folder. It is yours to edit, and `init` never overwrites it. Claude Code
reads `CLAUDE.md` instead; a `CLAUDE.md` containing the line `@AGENTS.md`
makes it read the same instructions.

## Steps are code

`useful-views run` and `useful-views sync` execute every step in `recipes/`
as an ordinary Python program, with your permissions, your environment
variables (including any passwords or tokens in them) and your network
access. There is no sandbox. An assistant should read the recipes, and ask
you, before running `run` or `sync` in a project or on changes it did not
write: a cloned repository, a pull request, a downloaded project.

These never run a step: `check`, `rules` and `cookbook --check`, which only
read files, and `cookbook`, which writes only `cookbook.json`.

## What to tell an assistant

- Read the project's `README.md`, `recipes/*/recipe.toml` and the catalog
  `useful-views/README.md` first. Never edit `datapackage.json`,
  `useful-views/README.md` or `cookbook.json`; useful-views writes them. Never
  put files in `raw/` or `useful-views/` by hand.
- Start with `useful-views rules` and `useful-views check --json`. `check`
  lists every problem, including every rule `run` or `sync` would refuse, and
  every recipe `sync` would run, with the reason.
- To add a recipe: `useful-views new-recipe <name>`, then fill in its
  `recipe.toml` and replace the stub step. Paths are the same strings in
  `recipe.toml` and in the step's code, relative to the project root, with
  forward slashes, and with no wildcards.
- To build: `useful-views sync`. It refuses before changing anything if any
  recipe breaks a rule, and lists every problem at once.
- After editing the code inside a step, after a `git pull` or a branch
  switch, or after replacing a raw file under the same name, `sync` alone
  does not notice: use `useful-views sync --all`.
- After `run <recipe>`, the recipes that read its views are out of date; the
  warning names them. Prefer `sync`.
- Run one command at a time. A second `run`, `sync` or `cookbook` started
  while another is working on the same project is refused ("another
  useful-views command is already changing this project"); wait, then run
  it again.
- After a sync, update the ledger: `useful-views cookbook`. In CI,
  `useful-views cookbook --check` checks it without needing the data; see
  [Cookbook](../concepts/cookbook.md).

## Machine-readable output

Every command takes `--json`, including `--help` and `--version`. Standard
output then carries exactly one JSON document, on success and on failure;
everything else, including the steps' own output, goes to standard error.
Every document has `ok` and a `warnings` list.

```sh
useful-views check --json
# {"ok": false, "findings": [{"kind": "out-of-date", "path": "recipes/b", "message": ...,
#   "rule": null, "recipe": "b", "reason": "upstream"}], "warnings": []}

useful-views sync --json
# {"ok": true, "removed": [], "dropped": [], "deleted": [], "ran": ["a", "b"],
#  "reasons": {"a": "changed", "b": "upstream"}, "warnings": []}

useful-views run pwt_panel --json
# on failure:
# {"ok": false, "error": {"type": "StepFailed", "message": ..., "recipe": "pwt_panel",
#   "sync_result": null, "step": ..., "returncode": 1, "stderr": ...}, "warnings": []}
```

Every `error` has `type`, `message`, `recipe` and `sync_result` (what `sync`
had done before it stopped, or `null`), plus facts for its type:
`RecipeRefused` has `rule`, `rule_title`, `path` and `refusals`;
`StepFailed` has `step`, `returncode` and `stderr` (at most its last 64 KB);
`OutputError` has `paths` and `problems`; `CookbookError` has `findings`;
`ExampleError` has `kind`.

| Exit code | Meaning |
|---|---|
| 0 | success |
| 1 | `check` found problems, or `cookbook --check` found the cookbook out of date |
| 2 | usage error: unknown command or option, or a bad `-C` path |
| 3 | refused or failed: the message says what to fix, and in which file |
| 130 | interrupted (Ctrl-C) |
| 143 | terminated (SIGTERM, from a CI cancel or a tool's time limit); 129 for SIGHUP, 149 for Ctrl-Break on Windows |

`-C <path>` runs a command in another folder, like `git -C`, so an assistant
whose working directory resets between commands does not need `cd`.

From Python, every exception carries the same facts as attributes and
`to_dict()`.

## For the assistant's own context

The documentation site publishes:

- an Agent Skills [`SKILL.md`](../../skill.md) on how to work in a
  useful-views project;
- [`llms.txt`](../../llms.txt), an index of the Python API, and
  [`llms-full.txt`](../../llms-full.txt), the whole documentation as plain
  text;
- the [command-line reference](../reference/cli.md), the
  [rules](../reference/rules.md) and the
  [check findings](../reference/check-findings.md).



## Reference

### Command line

<!-- Generated by scripts/generate_docs.py. Do not edit by hand. -->

Turn raw data into documented derived tables (useful views). Each recipe in recipes/ turns declared inputs into declared views by running numbered Python steps, and every run is recorded in datapackage.json.

Every command also runs as `python -m useful_views ...`.

```text
usage: useful-views [-h] [--version] [-C PATH] [--json] command ...
```

## Global options

These go before the command; `-C` and `--json` may also go after it.

| Option | Description |
|---|---|
| `-h`, `--help` | show this help message and exit |
| `--version` | show program's version number and exit |
| `-C` `PATH`, `--project` `PATH` | run as if started in PATH (like git -C), for example a project folder |
| `--json` | print the result as JSON on standard output; other output goes to standard error |

## `useful-views init`

Create the project structure in PATH: raw/, recipes/, work/, useful-views/, a starter README.md, AGENTS.md (for AI coding assistants), .gitignore and datapackage.json. Never overwrites a file.

```text
usage: useful-views init [-h] [-C PATH] [--json] [--force] [path]
```

| Argument | Description |
|---|---|
| `PATH` (optional) | project folder (default: the current folder) |

| Option | Description |
|---|---|
| `-h`, `--help` | show this help message and exit |
| `-C` `PATH`, `--project` `PATH` | run as if started in PATH (like git -C), for example a project folder |
| `--json` | print the result as JSON on standard output; other output goes to standard error |
| `--force` | create the project even in your home folder or at the top of a drive |

## `useful-views example`

Download an example project (raw data and recipes) from the useful-views-data repository, check every file's SHA-256, and set it up like init. Then run: useful-views sync

```text
usage: useful-views example [-h] [-C PATH] [--json] name [path]
```

| Argument | Description |
|---|---|
| `NAME` | the example to download, such as pwt |
| `PATH` (optional) | folder to create (default: <name>-example) |

| Option | Description |
|---|---|
| `-h`, `--help` | show this help message and exit |
| `-C` `PATH`, `--project` `PATH` | run as if started in PATH (like git -C), for example a project folder |
| `--json` | print the result as JSON on standard output; other output goes to standard error |

## `useful-views new-recipe`

Create recipes/NAME/ with a template recipe.toml and a stub step, 01_first_step.py. Both say TODO and are refused until you fill them in.

```text
usage: useful-views new-recipe [-h] [-C PATH] [--json] name
```

| Argument | Description |
|---|---|
| `NAME` | the recipe's name: lowercase letters, digits and underscores, like pwt_panel |

| Option | Description |
|---|---|
| `-h`, `--help` | show this help message and exit |
| `-C` `PATH`, `--project` `PATH` | run as if started in PATH (like git -C), for example a project folder |
| `--json` | print the result as JSON on standard output; other output goes to standard error |

## `useful-views run`

Run one recipe. It is checked against the rules first (useful-views rules), then its views are deleted and work/RECIPE/ emptied, each step runs in numeric order as its own process with this Python and the project root as working directory, every declared view must be written, and the run is recorded in datapackage.json and useful-views/README.md. Other recipes are not run.

```text
usage: useful-views run [-h] [-C PATH] [--json] recipe
```

| Argument | Description |
|---|---|
| `RECIPE` | the recipe's name, as in pwt_panel or recipes/pwt_panel/ |

| Option | Description |
|---|---|
| `-h`, `--help` | show this help message and exit |
| `-C` `PATH`, `--project` `PATH` | run as if started in PATH (like git -C), for example a project folder |
| `--json` | print the result as JSON on standard output; other output goes to standard error |

## `useful-views check`

Report what is missing or out of place, every rule a recipe breaks (refused), and recipes that sync would run. Changes nothing. Does not notice changes inside a step's code.

```text
usage: useful-views check [-h] [-C PATH] [--json]
```

| Option | Description |
|---|---|
| `-h`, `--help` | show this help message and exit |
| `-C` `PATH`, `--project` `PATH` | run as if started in PATH (like git -C), for example a project folder |
| `--json` | print the result as JSON on standard output; other output goes to standard error |

## `useful-views sync`

Make the views and datapackage.json match the recipe folders, like uv sync. Checks every recipe first and changes nothing if any would be refused. Then removes the records and view files of deleted recipes and outputs, and runs, upstream first, every recipe that is new, changed or missing its views, plus everything downstream of those.

```text
usage: useful-views sync [-h] [-C PATH] [--json] [--all]
```

| Option | Description |
|---|---|
| `-h`, `--help` | show this help message and exit |
| `-C` `PATH`, `--project` `PATH` | run as if started in PATH (like git -C), for example a project folder |
| `--json` | print the result as JSON on standard output; other output goes to standard error |
| `--all` | run every recipe again, upstream first (sync does not notice edits to a step's code) |

## `useful-views cookbook`

Write cookbook.json at the project root: every recorded recipe, in run order, with its description, inputs, outputs and steps, and the raw inputs the recipes read. Generated from datapackage.json, so the project must be in sync to write it. Commit it, like a lock file.

```text
usage: useful-views cookbook [-h] [-C PATH] [--json] [--check]
```

| Option | Description |
|---|---|
| `-h`, `--help` | show this help message and exit |
| `-C` `PATH`, `--project` `PATH` | run as if started in PATH (like git -C), for example a project folder |
| `--json` | print the result as JSON on standard output; other output goes to standard error |
| `--check` | do not write; exit 1 if cookbook.json is missing or out of date, or a recipe changed without a sync. Needs no data, so it works in CI |

## `useful-views rules`

List the numbered rules. A refusal names the rule it broke.

```text
usage: useful-views rules [-h] [-C PATH] [--json]
```

| Option | Description |
|---|---|
| `-h`, `--help` | show this help message and exit |
| `-C` `PATH`, `--project` `PATH` | run as if started in PATH (like git -C), for example a project folder |
| `--json` | print the result as JSON on standard output; other output goes to standard error |

## Exit codes

| Code | Meaning |
|---|---|
| `0` | success |
| `1` | check found problems, or cookbook --check found the cookbook out of date |
| `2` | usage error (unknown command or option, or a bad -C path) |
| `3` | refused or failed (the message says what to fix, and in which file) |
| `130` | interrupted (Ctrl-C) |
| `143` | terminated (SIGTERM, from a CI cancel or a tool's time limit; 129 SIGHUP; 149 Ctrl-Break) |


### Check findings

<!-- Generated by scripts/generate_docs.py. Do not edit by hand. -->

`useful-views check` (Python: `check()`) changes nothing and reports these kinds of finding, in this order. Each finding's message says what to do about it. With `--json`, each finding is an object with `kind`, `path`, `message`, `rule` (for `refused`, the rule's number), `recipe` (the recipe concerned, if any) and `reason` (for `out-of-date`: `changed`, `outputs-changed` or `upstream`), and the document also has a `warnings` list. The command exits with 1 when it finds anything.

| Kind | Meaning |
|---|---|
| `unreadable-recipe` | A recipe folder whose recipe.toml is missing or cannot be read. |
| `refused` | A recipe breaks a rule, so run and sync would refuse it. The finding names the rule. |
| `missing-input` | An input a recipe.toml declares is not on disk. |
| `never-run` | A recipe that has never run successfully. |
| `out-of-date` | A recipe that sync would run again: its recipe.toml or steps changed since it last ran, its outputs changed, or a view it reads will be made again. |
| `missing-recipe-folder` | datapackage.json records a recipe whose folder is gone. sync removes the record. |
| `missing-view` | A recorded view whose file is missing. |
| `stale-cookbook` | cookbook.json does not match the recorded recipes. Update it with useful-views cookbook. |
| `stale-catalog` | useful-views/README.md does not match datapackage.json (edited by hand, or left from a merge). useful-views sync writes it again. |
| `stray-file` | A file or folder in useful-views/, other than README.md, that no recipe makes. |

`check` does not notice changes inside a step's code or to a file's contents.


## How steps run

Steps are code: `run` and `sync` execute every step in `recipes/` as you,
with your permissions, your full environment (including any passwords or
tokens in environment variables) and your network access, with no sandbox.
Read the recipes of a project you did not write before running it. `check`,
`rules` and `cookbook` never run a step.

Each step runs as its own process:

- with the same Python interpreter as useful-views (`sys.executable`), so the
  libraries a step imports must be installed in that environment;
- with the project root as its working directory;
- with no standard input, so `input()` and `breakpoint()` fail;
- with `PYTHONIOENCODING=utf-8` and `PYTHONUNBUFFERED=1` added to the
  environment, so its output appears as it runs, in any terminal or notebook;
- in UTF-8 mode (`PYTHONUTF8=1`), so `open()` and `read_text()` read and
  write UTF-8 by default on Windows as on macOS and Linux. Pass `encoding=`
  to read a file in another encoding.

A step that fails with `ModuleNotFoundError` gets a message naming the
interpreter and the command to install the missing package into it.

## Stopping a command

Ctrl-C stops the running step, removes the views of the recipe it belongs
to, and exits with 130. On the command line, SIGTERM and SIGHUP (sent when a
CI job is cancelled, when a tool's time limit runs out, or when a terminal
closes) do the same, exiting with 143 and 129; on Windows, Ctrl-Break exits
with 149. A signal that was already being ignored stays ignored, so
`nohup useful-views sync &` keeps running after you log out. When `sync` is
stopped, its message says what it had already done and which command to run
next.

`run`, `sync` and `cookbook` hold a lock on the project while they work, in
`work/.useful-views.lock`, so a second one started meanwhile is refused
instead of overwriting the first one's records. The operating system
releases the lock when the command ends, however it ends.

## Environment variables

| Variable | Effect |
|---|---|
| `USEFUL_VIEWS_DATA_URL` | Where `example` downloads from, instead of the pinned tag of the useful-views-data repository: a mirror's URL, or a local copy (a folder path or `file://` URL) laid out like the repository (`examples/<name>/...`). Downloads are still checked against the pinned SHA-256 checksums. Useful without internet access. |
| `HTTPS_PROXY`, `SSL_CERT_FILE` | Read by Python when `example` downloads: a proxy to go through, and a certificate file for networks that inspect encrypted traffic. |

## Output

- Human-readable results go to standard output; warnings, progress lines and
  error messages go to standard error.
- With `--json`, standard output carries exactly one JSON document, also on
  errors (`{"ok": false, "error": {...}}`); everything else, including the
  steps' own output, goes to standard error. The JSON is ASCII-escaped, so it
  stays valid whatever the console's encoding.

The exit codes are listed in the [command-line reference](cli.md#exit-codes).


### Rules

<!-- Generated by scripts/generate_docs.py. Do not edit by hand. -->

`run` and `sync` refuse a recipe that breaks one of these rules, before changing anything on disk, and `check` reports it as `refused`. A refusal names the rule; from Python, `RecipeRefused.rule` is its number. On the command line: `useful-views rules`.

## Rule 1: recipe.toml is complete

recipe.toml exists and can be read. It has a one-sentence description that does not start with TODO, and inputs and outputs, each a list of at least one path with no path listed twice.

## Rule 2: Steps are numbered and described

The recipe has at least one step file named NN_name.py. Each starts with a # comment saying what it does, not starting with TODO, and no two steps share a number.

## Rule 3: Steps are Python files

No file in the recipe folder is named like a step in another language (.R, .Rmd, .ipynb, .qmd, .jl, .do).

## Rule 4: Inputs exist and are allowed

Each input is a file under raw/ or directly inside useful-views/, written relative to the project root with forward slashes and no wildcards, with names that can exist on Windows, and exists with exactly that letter case. A raw input is not empty and can be read.

## Rule 5: Input views come from a recipe that has run

An input inside useful-views/ is an output of another recipe, and that recipe has run successfully.

## Rule 6: No cycles

The recipe does not read its own view, directly or through a chain of other recipes.

## Rule 7: Outputs are allowed and unique

Each output is a .parquet file directly inside useful-views/, with a name that can exist on Windows, is not useful-views/README.md, and is not made by another recipe.

## Rule 8: Resource names are unique

No two recorded files share a resource name: a view's file name, or a raw file's path below raw/, without the extension, lowercased, with '/' and other characters outside a-z, 0-9, '.', '_' and '-' turned into '-' (accented letters lose their accents; letters of other scripts become their Unicode code point).

A recipe name that does not match a folder in `recipes/`, or a folder name that is not allowed, is also refused, with `rule` set to `None`.