Forty minutes. A note app, a browser, and a free chat if you already have one. You do not need an API key, an email signup, or an upload. A mechanism here means the approach: plain chat, retrieval (putting a document in the request), an agent with a named tool, or fine-tuning (more training).
10 minutes: three jobs
For each job, write the approach, what would prove you wrong, and what you will not do first. Write your answers before you look at the sample set below.
Job A. People ask questions that live in a 40-page handbook. The handbook changes. They paste last year’s PDF and then argue.
Job B. A refund has to land in the billing system for the amount on the order. A wrong refund is a real debit.
Job C. Short replies drift off the macros. You have about forty good ones and forty bad ones, labeled by a person.
One defensible set, not the only set: A is retrieval or a pasted section. B is a tool your code runs, with a human or a hard rule on the large ones. C is a prompt with a few good replies first, and a fine-tune only if the held-out replies still miss. For A, try a question whose answer you withheld. A confident reply may mean the model is drawing on general training rather than your document.
15 minutes: tokens are not the same as words
- Paste this into a note and count the words: “Please refund order 1844, forty-two dollars and ten cents, dated 2 March.” A token is a chunk of text the model counts. It is often smaller than a word.
- Open the OpenAI tokenizer and paste the same sentence. Write down the token count it shows and the date.
- Compare the tokenizer count with a word count. They measure different units and often differ. Do not convert either one into a price. If a product shows a usage meter, that meter is the count that matters.
15 minutes: the paragraph you withheld
In any free chat you already use:
- Ask: “What is our shipping refund rule?” Do not paste a policy.
- Then paste only this invented line, clearly marked as a drill: “DRILL POLICY, not a real company: shipping is refundable within 10 days, and not after.”
- Ask again.
The first reply can sound sure and still be a guess. The second should be able to quote the 10 days, because you just handed it the sentence. If it still cannot, check whether the product included your pasted text in the request. That is a finding.
Write four lines: what the first reply did, what the second did, which mechanism that is (chat versus retrieval you performed by hand), and the date.
Sources & limits: The drill policy is fake on purpose. Do not file it as your company’s rule. If the tokenizer page has moved, say so in your note and skip the number. A dead page is not a guess. This lab does not collect an email or an upload.