Script or Agent?

a fun project · not a product

Before the autopsy

The 3×3 Rule

Nine yes-or-no checks for whether AI adds value here, or whether you are about to build a demo. No model in this check. The count is the method.

If you cannot clearly answer problem, user, and outcome, it is not the time to apply AI yet. Define the case first.

Define the case

Name the pain, who feels it, and how you will know it worked. The first three are the gate. The yes/no count below does not replace them.

The nine checks

Yes or no. 7–9 yes: a good candidate. 4–6: a scoped pilot. 0–3: do not apply AI yet.

1

Is there scattered text or information to understand?

Searching, summarizing, classifying, extracting, drafting. Clear rules and little ambiguity are a job for rules.

2

Does the knowledge change often?

If it changes often, retrieval keeps answers current. If it is static, a template or a rule stays righter for longer.

3

Does the task occur many times per week?

High repetition is where a model’s cost amortizes. A rare task has a marginal payoff.

4

Do we have curated, traceable data?

Aim for 20–50 relevant items, with a date or version, and a consistent shape. If not, prepare them before you build.

5

Can we measure time, errors, or CSAT?

One or two simple metrics, agreed with the business. No metric, no improvement.

6

Is the error recoverable?

Recoverable mistakes can be reviewed. Zero-error work needs a human in the loop, or no generative model.

7

Is the decision reversible?

Irreversible or high-cost outcomes need approval. Reversible, low-cost ones can be more autonomous.

8

Is seconds-level latency acceptable?

A model call is the right shape only when a few seconds is fine.

9

Is the cost per use lower than the estimated savings?

High cost and low savings is not worth it. Low cost and high savings is.

Two gates

These sit outside the nine. A rule that already works beats a model. Money or legal impact needs a human.

Can this be solved with a clear rule or a small process change?

If yes, use the rule. This gate sits outside the nine and it wins.

Is the impact high — money, legal, or hard to undo?

High impact means a human approves before anything is sent or committed.

How to read the bar

When AI adds value, and when it does not

It adds value when it…

  • Reduces time or errors in information work: searching, summarizing, classifying, extracting, drafting.
  • Gives a clear answer at the exact moment in the flow where it is needed.
  • Supports an operational decision with relevant, understandable information.
  • Can be measured with one or two simple metrics agreed with the business.

It does not add value when…

  • A simple rule or a process tweak already solves it.
  • There is no reliable data, and it cannot be prepared soon.
  • The cost of a mistake is high, and nobody reviews the result.
  • There is no success metric, or no owner for the outcome.
Decision basis — the minimum
  • Case and metric. One case, one or two KPIs. Without a metric, there is no improvement.
  • Curated data. 20–50 relevant items, with a date or version, and a consistent shape.
  • Guardrails. Clear limits on content and actions. A human reviews where errors are costly.
  • Simplicity. A clear interface and fewer steps. That is what gets adopted.
  • Observability. Log usage, cost, latency, and quality so the next change is evidence.

If it misses the bar, the answer is not a hard no. It is a not-yet. Ship, iterate, or pause.

Levels: luxury, useful, necessary
  • Luxury. Nice-to-have. An aesthetic or comfort upgrade with no clear effect on time, errors, or demand.
  • Useful. Should-have. It saves time or reduces errors, and the process can live without it.
  • Necessary. Must-have. Without it, the process does not scale or it misses the service level.
What actually decides it
Information complexity
Low, with clear rules: rules win. Medium or high, with natural language and several sources: a model can help.
Knowledge variability
Static knowledge belongs in a template. Knowledge that changes belongs in retrieval.
Scale and frequency
Low demand, marginal benefit. High repetition is where the cost amortizes.
Error tolerance
Zero errors means a human in the loop, or no generative model. Recoverable errors can be an assistant plus review.
Decision reversibility
Irreversible or costly: more controls and an approval. Reversible and cheap: more autonomy.
Data readiness
Incomplete data gets curated first. Curated, traceable data is where a model behaves.
Measurability
No metric is innovation theater. A clear KPI is an evidence-based decision.
Relative cost
High cost per use and low savings is not worth it. Low cost and high savings is.
Coordination and cognitive load
High friction between teams and systems is where summaries, search, and normalization help. Low friction is marginal.
The decision tree
  1. Can it be solved with rules? Use rules.
  2. Does it involve text or information — searching, summarizing, classifying, extracting, drafting? Then a model can help.
  3. Does the knowledge change frequently? Use retrieval to keep it current.
  4. High impact, money or legal? A human approves.
  5. Can you measure the change? If not, define the metric first.

Less magic, more method: a clear problem, a simple metric, ready data, defined limits, and continuous improvement. If those are not in place, do not apply AI yet.