Measuring Impact and Leading Change

"We saved a lot of time" is the most common claim about AI and the hardest to prove. Without measurement, you cannot know whether the effort is worth it, which uses to keep and which to drop. In this lesson you will learn to measure before and after, to count the real costs, to avoid classic traps, and to guide colleagues through the change with honesty.

You have rules and workflows. The question now is whether all of this actually helps, and how to bring other people along. Measurement and change management are what turn a few enthusiasts' experiments into lasting improvement.

Camille's director asks for proof

Three months after the first experiments, Camille's director says: "I hear people like these tools. What is the result for the company?" Camille realizes she has impressions but no figures: "I think we save time" will not convince anybody. She decides to build a simple measurement method, which she applies to the use cases chosen in chapter 1.

Measure before and after

The principle is simple: take a baseline (how things are without the tool), then measure again with the tool, under comparable conditions. Without a baseline, a gain is a feeling.

What to measure

Dimension	Examples of indicators	How to collect
Time	Minutes to complete a task, turnaround time for a request	Timing a sample of tasks, timestamps in your tools
Quality	Number of corrections needed, errors found at review, customer complaints, rating on a quality grid	Review grid applied to a sample, error log
Satisfaction	How users and recipients feel about the work or the output	Short surveys, interviews, customer feedback
Volume and capacity	Number of requests handled, backlog size	Existing counters
Risk and compliance	Incidents, near-misses, rule violations	Incident log, audits

Choose two or three indicators per use case, not twenty. Measure what you can really collect with little effort. And include at least one quality indicator: time saved on a badly done job is not a gain.

A simple protocol

1. Choose a task and define precisely what counts as "done".
2. Before: time ten occurrences without AI, and record quality (for example, how many corrections a colleague makes).
3. After: time ten occurrences with AI, including the time to prepare the prompt, read, check and correct.
4. Compare time, quality and effort. Note anything unusual (a particularly difficult case, a new person).
5. Repeat after a few weeks, because novelty and learning effects change results.

We do not give you target numbers: results depend on the task, the person, the tool and the quality of your prompts. The only figures that matter are your own, measured honestly.

A ready-to-use prompt: design the measurement

Help me design a simple before/after measurement for this AI use case: [use case, e.g. drafting replies to supplier emails with an approved assistant].
Context: [team size, how often the task happens].
Propose: (1) 2 or 3 indicators (time, quality, satisfaction) and how to collect each with little effort; (2) a protocol for the "before" and "after" measurements, including how many samples to take; (3) a short quality grid with 4 criteria rated from 1 to 3; (4) the biases that could distort the comparison, and how to limit them; (5) the decision rule: what result would make me keep, adjust or stop the use case.
Do not invent target figures or expected gains.

Count the real costs

A fair assessment includes costs, not only gains:

• Subscriptions and licenses, including those for tools that quietly accumulate.
• Time to prepare and write prompts, and to maintain the library.
• Review and correction time, often underestimated.
• Training and support: sessions, documentation, time to help colleagues.
• Setup and integration: connecting tools, building workflows, testing.
• Risk management: policy writing, legal advice, security reviews.
• Hidden costs of errors: a corrected customer message, an incident, a lost relationship.

Compare the total with the value of the time and quality gains, and remember that time saved is only valuable if it is used for something useful: freed hours that disappear into more of the same work do not show up in results.

Traps to avoid

Rework

The assistant produces something fast, but you spend a long time fixing it. If you only measure the time to generate, you overestimate the gain. Always measure end to end, from the start of the task to a validated result.

The showcase effect

A successful demo, an impressive example or a quick win is not a measurement. Enthusiasts also tend to report their best cases. Measure on ordinary tasks, including difficult ones, and ask for honest feedback, including from skeptics.

Shifted effort

A task becomes faster for you but slower for someone else: the reviewer now receives more text, the customer receives longer emails, the manager has more documents to read. Look at the whole chain.

Quality erosion

Output that is "good enough" may slowly become generic, careless or error-prone because nobody checks properly. Keep your quality indicators.

Counting hours as money

Ten minutes saved by ten people is not automatically a salary saved. Present the result as capacity or quality gained, and be explicit about what was done with it.

Leading the change

Technology adoption is mostly a human matter. People have understandable concerns: will this replace me, am I being monitored, will I look incompetent, will I be blamed for mistakes? Ignoring these feelings is the fastest way to resistance or to hidden use.

• Start with real problems: begin with tasks people find tedious, not with the technology.
• Involve people early: let colleagues choose use cases and shape rules. Where required, inform and consult employee representatives about tools that affect working conditions.
• Be clear about intent: say what you are trying to achieve and what you are not (for example, that AI will not be used to monitor individual performance, if that is the case).
• Train and support: short practical sessions, a place to ask questions, champions in each team.
• Share failures as well as successes: a "what went wrong" moment teaches more than a demo.
• Respect different paces: some people will want to go further; others will need time or to opt out of certain uses.
• Communicate regularly: share measured results, updates of the rules and lessons learned.

Communicating results honestly

A good update to your team or management can be short: what we tested, how we measured, what we found (time, quality, satisfaction), what it cost, what surprised us, what we will do next. Include limits ("ten samples", "one person", "three weeks"). Honest, modest reporting builds more trust than an enthusiastic claim.

Bad use versus good use

Bad use	Good use
"We saved a lot of time" without figures.	Before/after timing on samples, including the checking time, with a quality grid.
Rolling out to everyone after one successful demo.	A pilot with a few volunteers, honest feedback and adjustments before widening.
Announcing that AI will increase productivity and expecting people to follow.	Listening to concerns, explaining intent, training and showing results from your own context.

Your turn: your measurement sheet

Exercise. For each of your three first use cases, fill in a measurement sheet: indicators (time, quality, one other), the "before" method, the "after" method, the costs to count, and the decision rule (keep, adjust, stop). Do the "before" measurement for at least one case this week. Then write a five-line message to your team or manager announcing the experiment honestly, including what you will not do.

See a solution

Camille's sheet for "draft replies to supplier emails": Indicators: minutes per reply (time), number of corrections made by Samir on a sample of ten replies (quality), and Samir's own rating of effort from 1 to 5 (satisfaction). Before: she times ten replies written without AI. After: ten replies with the assistant, timing from opening the email to sending, including checks. Costs: license share, prompt preparation, training time. Decision rule: keep if time is lower and corrections are not higher; adjust if time is lower but quality lower; stop if neither improves after a month.

Her message: "For the next four weeks, three of us will test an approved assistant to draft supplier replies. We will time tasks and review quality, and we will share the results, whatever they are. Nothing will be sent without being read by a person, and the data does not include personal information. This is not about monitoring individual performance."