Measure what happened, not what the assistant generated.
A useful scorecard connects conversation activity to verified next steps. It also makes skipped records, failed handoffs and incomplete evidence visible instead of burying them in one success number.
Define the denominator before the result
Specify the account, date range, lead sources and eligible conversation set. Paused officers, excluded stages, stop requests and records already handled by a person should not automatically be counted as unanswered assistant failures.
Keep distinct outcomes distinct: generated reply, accepted send, delivered message, borrower response, recorded appointment, attended consultation, application and funded loan. One event does not prove the next. If delivery or attendance evidence is missing, label it unknown rather than silently treating it as success.
Build a small evidence-led scorecard
| Measure | Evidence | Caution |
|---|---|---|
| Coverage | Eligible inquiries and unresolved cases | Separate exclusions and incomplete history |
| Response speed | Inquiry time to a defined relevant response event | Show slow cases as well as an average |
| Conversation quality | Context use, relevance and human overlap | Review a consistent sample, including failures |
| Appointments | Verified bookings and separately verified attendance | Do not count a link or suggested time as booked |
| Economics | Complete costs for the same period and scope | Disclose bundled charges and omitted labor |
Avoid unfair before-and-after claims
A new campaign, different lead source or expanded team can change outcomes independently of the assistant. Compare equivalent scopes and disclose remaining differences. A few promising conversations can guide a pilot but do not establish a conversion-rate improvement for every officer.
Read the exceptions: a repeated question can harm quality even when a text was delivered. A lower subscription bill can still be expensive if the officer must repair every handoff. Check outcomes alongside contact boundaries, opt-outs and the time required for human review.
Keep a review record people can reproduce
Document the period
Use one date range for expenses, eligible inquiries and outcomes.
Retain outcome evidence
Keep the relevant send, appointment and attendance records within approved systems.
Record uncertainty
Show missing delivery or attendance evidence instead of filling gaps with estimates.
Review changes
Compare the same definitions after configuration or workload changes.