Compare Two Assistants On One Real Job

A Comparison With Practical Value

Compare two assistants on one real job, using the same source, finish line and correction budget. Record whether the work succeeds and how much human help it needs.

Use a real task or start with the fictional meeting example. No private information, connection or external action is needed for the example.

Keep The Test Fair

1. Name one job. For this example: extract the assigned actions from supplied meeting notes. This is a Transform task.
2. Write the finish line. All three assigned actions, correct owners and deadlines, no invented details, no unassigned FAQ action, and only the requested table.
3. Keep the starting conditions the same. Start fresh chats, paste the same text and record each assistant’s product, model or mode, plan, tools and date. Note any saved preferences that affect the result.
4. Set a correction budget. For this exercise, allow up to two follow-ups to fix specific failures. Use the same limit for both. If it still fails, record the failure and the time spent.
5. Check before accepting. Compare with the source and expected result below. Record first-pass success, follow-ups, active human time and elapsed time. Include manual edits.
6. Repeat before judging reliability. Use fresh chats and several representative examples. Report counts, such as four passes in five attempts. A small trial informs a choice for the tested task; it does not prove general reliability.

Required clarifications are useful when a task is ambiguous. This example is deliberately explicit, so fidelity can be tested without ambiguity about the intended outcome.

Give Both Assistants The Same Job

Copy the complete prompt. Do not improve it for one assistant after seeing the other’s answer.

Turn the meeting notes below into a table with three columns: Action, Owner, Deadline.
Include only actions with an explicitly assigned owner. Preserve the deadlines as written. Do not invent dates, owners or extra work. If a required detail is missing, mark it as not stated. Return only the table.

Meeting notes:
Mary: Send the revised budget by Thursday.
Tom: Confirm supplier availability by Friday.
Gloria: Update the launch checklist by Monday.
Peter: A customer FAQ could be useful later.
Meeting note: No owner or deadline was agreed for the FAQ.

Check the meaning, not whether the wording is identical. The three owners and day names must match. Do not accept a beautifully formatted fourth row for an unassigned task.

Keep The Evidence Beside The Result

Use this compact record alongside the test notes. Keeping the failures prevents a fast unsuccessful attempt from looking like a bargain.

Task and completion criteria:
Source, language and constraints:
Test date:

Assistant A — product / model or mode / plan / tools / saved preferences:
Assistant B — product / model or mode / plan / tools / saved preferences:
Correction limit: two follow-ups each

For each assistant and each attempt, record:
- Usable on the first response? Yes / No
- Usable within the correction limit? Yes / No
- Follow-ups and manual edits:
- Active human minutes (prompting, reading, checking, correcting, finishing):
- Elapsed minutes from start to a usable result or stopping the attempt:
- Missing, incorrect or invented details:
- Unrequested extra work:
- Optional work deliberately pursued after the result was usable:

Across attempts:
- First-pass successes / total attempts:
- Successes within the correction limit / total attempts:
- Failed attempts and their time:
- What these results support — and what remains untested:

For creative work, replace the exact table check with criteria agreed in advance: tone, length, factual fidelity and suitability for the audience. A faster draft is not automatically a better one.

What Does The Evidence Support?

The exercise is complete when the evidence answers these questions:

• Which assistant met the same finish line, and how often?
• Which required less checking, correction and manual finishing?
• Which was faster in elapsed time?
• What conditions were tested, and what remains unknown?