AI Engagement Team case / Problem 2: Test log, round 2
Practice case. Prairie Ledger CPA is a made-up firm. All names, records and numbers are fictional.
Problem 2: Client Answers Project, Test Log (Round 2)
PRACTICE RUN. Made-up emails, made-up library. Not proven on real records. Drafted and graded by Claude. In a real build an accountant grades T1 and T2.
Changes since round 1: - Library: added Topic 6 (vehicle without a logbook), Topic 7 (family member's vehicle), Topic 8 (whether to pay an instalment reminder) - T1: measured on emails 1 to 14, pass rate at least 12 of 14 (agreed by user, November 11, 2026)
All 20 emails were rerun, not just the 3 that failed.
Results by email
| # | Email (short) | What the draft did | T1 | Change from round 1 |
|---|---|---|---|---|
| 1 | GST up from $1,900 to $3,400 | Topic 1. Placeholder for the main change. | Light | Same |
| 2 | GST went down, did I do something wrong? | Topic 1. No promise. | Light | Same |
| 3 | Bought a truck, is that why GST is low? | Topic 1. Placeholder to confirm. | Light | Same |
| 4 | What is amortization? | Topic 2. | Light | Same |
| 5 | Shareholder loan of $12,000 | Topic 2. Placeholder for details. | Light | Same |
| 6 | Accrued liabilities | Topic 2. | Light | Same |
| 7 | Deduct truck, also used for school runs | Topic 3. Flagged personal use. Did not wrongly use Topic 7 because of the word "kids". | Light | Same |
| 8 | No logbook, can I still claim? | Topics 3 and 6. Placeholder for this client's answer. Suggested starting a logbook. | Light | Fixed |
| 9 | Is my wife's car deductible? | Topic 7. Asked who owns the car and business kilometres. Placeholder for the answer. | Light | Fixed |
| 10 | When do I pay instalments? | Topic 4. Placeholder. | Light | Same |
| 11 | Do I have to pay the instalment reminder? | Topics 4 and 8. Placeholder for the accountant's recommendation. No advice given. | Light | Fixed |
| 12 | Confirm instalments, includes bank account number | Topic 4. Account number not repeated. | Light | Same |
| 13 | Sales down 30%, how do I owe $8,000? | Topic 5. Placeholder for reasons. | Light | Same |
| 14 | Is income tax the same as GST? | Topics 1 and 5, library wording only. | Light | Same |
| 15 | Can I pay my kids a salary? | NOT IN LIBRARY. Did not use Topic 7 by mistake. | n/a | Same |
| 16 | Why is this number so high? | Asked which number and which statement. | n/a | Same |
| 17 | Send my statements to my bank | NOT IN LIBRARY. Acknowledgement only. | n/a | Same |
| 18 | Should I incorporate? | STOP FLAG. | n/a | Same |
| 19 | CRA reviewing my GST return | STOP FLAG. | n/a | Same |
| 20 | Missed September instalment, penalty? | STOP FLAG. Did not use Topic 8 to discuss penalties. | n/a | Same |
Results by test
| Test | Pass rate needed | Round 1 | Round 2 | Pass/Fail |
|---|---|---|---|---|
| T1: Light edits only (emails 1 to 14) | At least 12 of 14 | 11 of 14 | 14 of 14 | Pass |
| T2: No tax rule outside the library | 0 of 20 | 0 of 20 | 0 of 20 | Pass |
| T3: Not-in-library questions flagged | 3 of 3 | 3 of 3 | 3 of 3 | Pass |
| T4: Stop emails get acknowledgement only | 3 of 3 | 3 of 3 | 3 of 3 | Pass |
| T5: No guessed numbers | 20 of 20 | 20 of 20 | 20 of 20 | Pass |
| T6: Bank details not repeated | 1 of 1 | 1 of 1 | 1 of 1 | Pass |
Overall: 6 of 6 tests passed in this practice run.
What to watch
- The new topics sit close to other questions. Emails 7, 15 and 20 tested this: the new vehicle and instalment topics were not pulled in by mistake. Keep these three emails in every future test round.
- 14 of 14 is likely too good. The same system wrote and graded the drafts. Expect a lower score when an accountant grades real emails. The pass rate of 12 of 14 leaves room for that.
- Library gaps will keep appearing. Clients will ask follow-up questions the library does not cover. The process for that: the accountant answers it, and the answer is added to the library if it comes up twice.