Every draft looks confident. Some of them are wrong.
An assistant that writes support replies is not hard to build. Deciding which replies may go out without a person is the hard part, and the confidence score is a poor guide on its own: the most dangerous drafts in this queue are the ones that read best. Set the line, choose what always goes to a person, and clear the queue yourself.
SimulationThe queue, the drafts and the confidence scores are written for this demonstration, and the routing is real code running in your browser. Nothing is sent anywhere. The production build puts the same gate in front of a live assistant, records every override, and uses those overrides to retune the line.
A draft scoring at or above this goes out with nobody reading it. Lower the line and more goes out, faster, with more risk. Raise it and the queue slows down.
At a 80 line, with 3 rules in force, 2 of 8 drafts would be sent without a person, 4 would wait for a review and 2 would go straight to someone. None of the drafts sent without a person is one that needed one.
The queue
Eight tickets from this morning, in the order they arrived. The chip is what the policy would do with each one right now.
Marcus Bell
Where is my order? · email · waiting 12 minutes · calm
- Customer
Order 4417 was due Tuesday and tracking has not moved. Can you check?
08:41
The proposed reply
93Hello Marcus, thank you for flagging this. Order 4417 left our warehouse on Monday and the courier last scanned it at their Bristol depot on Tuesday. It is showing as in transit rather than lost, and the courier's own guidance is to allow two working days after the last scan before treating it as delayed. I will check again tomorrow morning and message you either way.
- Order 4417 tracking, refreshed 08:38
- Courier delay guidance, paragraph 3
A delivery commitment No rule covers this, so the score decides.
Sent without a person
Scored 93, at or above the 80 line.
What you decided
Nothing decided yet. Send, edit or escalate a ticket above.
What the demo is showing
Why a score is not enough
- Confidence measures how well the draft is written. It does not know whether the business should be saying it.
- The most dangerous drafts here score in the eighties: a refund asserted in the past tense, and a credit nobody authorised.
- A rule that sends money, legal and safety to a person catches them regardless of what the score says.
- Turn one rule off and the strip shows that draft cross the line immediately.
What the numbers mean
- Sent without a person is the automation rate, and the number to grow over time.
- Should not have gone is the cost of the line being too low.
- Overrides are how often a person disagreed with the machine. A high rate means the line is wrong in the safe direction, which is still wrong.
- The demo sends nothing. In production every sent reply and every override is recorded against the ticket.