Prefect is a Python-native workflow orchestrator founded on a deceptively simple philosophical pitch: most engineering work in data pipelines is not building features, it's preventing failure modes. This idea — which the Prefect team named "negative engineering" — is the lens through which the entire product was designed. Prefect is the orchestrator most loved by people who actually try it, has arguably the cleanest Python API in the category, and remains a smaller player than Apache Airflow or Dagster by mindshare. It is a good system that has not quite won the market it deserves to.

Origin Story: Jeremiah Lowin, the Airflow Insider Who Started Over

Prefect was founded in 2018 by Jeremiah Lowin, who was previously a committer on Apache Airflow and a member of the Airflow PMC (Project Management Committee). Lowin came from a quantitative finance background — he had been a managing director at AQR Capital Management — before getting deeply involved in the open-source data world. He spent enough time inside Airflow to understand its foundational design choices and conclude that several of them could not be patched. So he left the project and started Prefect with the explicit goal of redesigning workflow orchestration from scratch.

Prefect 1.0 ("Prefect Core") shipped in 2019. It was already a meaningful improvement over Airflow in API ergonomics, but it still inherited some of Airflow's mental model. In 2022, the team did something unusual for an open-source project with real adoption: they rewrote it. Prefect 2.0 (originally code-named "Orion") replaced the static-DAG model with a more dynamic, runtime-discovered approach. You write normal Python functions, decorate them with @flow and @task, and Prefect builds the graph as your code runs. This made Prefect feel less like "I am defining a config file" and more like "I am writing Python that happens to be orchestrated."

The company has raised over $80 million across multiple rounds from Tiger Global, Bessemer, and others. Prefect Cloud is the managed offering and the primary commercial product.

The "Negative Engineering" Pitch

The phrase negative engineering is Prefect's marketing crystallized into a concept, but it's actually a useful one. The argument:

When you write a data pipeline, the "happy path" code — read this CSV, transform it, write to Snowflake — is maybe 20% of the work. The other 80% is dealing with failure: retries, timeouts, idempotency, alerting, restart logic, partial-failure recovery, schedule drift, late data, dependency failures. None of this code adds business value. All of it is "negative engineering" — work you do to prevent bad things rather than to create good things.

A good orchestrator should do as much negative engineering for you as possible, so you can focus on the actual transformation. Prefect's design choices follow from this:

• Retries, timeouts, and caching are decorator arguments, not custom code. @task(retries=3, retry_delay_seconds=60).
• Failures are first-class. Tasks return State objects that explicitly model success, failure, retry, skipped, and cached states.
• Flows can be defined dynamically. You can have a for loop that creates tasks based on runtime data. (Airflow added "dynamic task mapping" much later, and it's still awkward.)
• Local execution is trivial. python my_flow.py just runs it. There's no scheduler-and-DB-and-webserver ceremony to get started.

Kitchen analogy: Negative engineering is everything the head chef does that isn't cooking — making sure ingredients arrived, the oven works, the dishwasher is empty, a backup plan exists if the fish doesn't show up. Prefect is the orchestrator that handles all of that infrastructure-of-the-kitchen so you can focus on the recipe. The recipe is small. The infrastructure is most of the job.