How to measure if security training works

The metric that proves security training works is not how many people finished the course, but whether those who used to fall for a scam stop falling when they are tested again weeks later. Everything else (courses completed, hours logged, clicks avoided in one campaign) measures activity, not outcome. If we run a security program and all we can report is completion, we are reporting that people opened the material, not that they changed how they behave against a real attack. That difference decides whether the budget we defend to leadership is buying security or buying peace of mind.

The conclusion: the metric that proves change is the retest, not completion

The retest is the only metric that tells apart someone who learned from someone who merely remembered an email for a while. It means testing the person again with an attack of the same type and difficulty, but in a different context, some time after the failure, to see whether they recognize the pattern or fall again. Completion, by contrast, confirms the material was consumed, and that is not the same as sustained behavior change.

This distinction has support. There is peer-reviewed evidence that completing training does not by itself predict a reduction in real failures (Ho et al., IEEE S&P 2025; Lain et al., IEEE S&P 2022). Put differently: a team can show one hundred percent course completion and still hand over credentials when the right email arrives. What demonstrates change is retesting the behavior, not stacking up certificates.

Why click rate and course completion mislead

Click rate and completion mislead because they measure the wrong moment and create a false sense of progress. A single campaign's click rate depends as much on the employee as on how obvious the simulation was: an easy campaign lowers the number and looks good in the report, without anyone becoming more resilient. And a low click rate can coexist with people who would click on a well-built pretext aimed at their role.

Completion has the same problem, made worse. In real studies, most employees engage with training material for a minute or less, so a course marked as finished can mean someone left the video running in another tab. Training that looks good on the dashboard but never touches behavior leaves the risk exactly where it was. The problem is not measuring; it is measuring the program's proof of life instead of its effect.

What a targeted retest is and how it is applied weeks later

A targeted retest is a second test sent to whoever failed, after a waiting period, using an equivalent but different attack from the original. The key lies in two details. First, timing: if the test arrives the same day as the training, it measures fresh memory, not learning; a retest that lands weeks later measures whether the lesson survived forgetting. Second, variation: if it is the same email, the person may recognize it by its form without understanding the pattern, so the retest changes the context while keeping the type of deception and the difficulty.

The retest result is what turns training into a measurable data point. If the person spots the attack, there is evidence of change. If they fall again, we know the module did not work for them and that they need a different approach, not another certificate. A single success is not enough either: resilience is confirmed when someone passes several tests in a row without falling, spread over time, not in one good answer.

Metrics that actually predict fewer real incidents

The metrics that actually anticipate fewer incidents are those measuring behavior under repeated pressure, not content consumption. The main one is the repeat-failure rate after retest: of the people who fell and received remediation, what share falls again on an equivalent test. A repeat-failure rate that drops over time is the closest signal to a real reduction in risk, because it measures exactly what an attacker would exploit.

Other useful metrics point the same way. Recovery time, meaning how many tests a person needs to return to a low-risk state after a failure, shows whether remediation works fast or does not work at all. Risk concentration, which areas or roles accumulate repeated failures, tells you where to put the effort instead of treating the whole company the same. And prior exposure matters: a person with credentials already leaked in a breach is a priority target, because the attacker already has half the job done. It helps to keep the framing in mind while measuring all of this. Cisco's 90-5-5 framework estimates that around 90 percent of breaches involve a human factor (source: Cisco), so the metric that moves the needle is the one proving that human factor became more resilient, not the one counting video hours.

How to read these metrics in a report for leadership

A report for leadership should answer a single question: are we less vulnerable today than a quarter ago, yes or no. That means leading with the repeat-failure trend after retest and with how risk concentration changed, not with the list of completed courses. Leadership does not need to know how many emails were sent; it needs to know whether the people who were vulnerable stopped being vulnerable and where the hotspots remain.

The common mistake is presenting volume as if it were outcome. A thousand simulations sent and ninety percent of courses finished fill a slide, but they do not answer the question. A single curve, repeat failure going down, says more than ten activity indicators. The practical rule for building that dashboard lies in how to assemble reports that truly inform a decision, not ones that only prove the program exists.

What to ask your platform to measure behavior change

You have to demand that the platform can prove change, not just record it. The concrete question for any vendor is: do you automatically resend an equivalent attack to whoever failed, weeks later, and show me whether they fell again? If the answer is that it delivers courses and measures completion, it is measuring activity. If it resends the same simulation the next day, it is measuring memory. Only a targeted, delayed retest measures learning.

It is worth asking for three more things. That remediation arrives at the moment of failure and is specific to the attack, not a generic video, because an immediate, short correction is the one people actually consume. That the personalization of tests is based on role, prior behavior and the real exposure of each person, not identical sends for everyone. And that the report clearly separates activity from outcome, so no one mistakes a busy dashboard for a safer team.

We built Fensivo around exactly that question. The human risk management (HRM) platform closes the loop: it detects exposed credentials, tests each person with personalized simulations, delivers immediate remediation on the failure and, weeks later, runs a retest with a different template of the same type to validate that behavior changed, not that an email was remembered. That is how we measure real resilience instead of content consumption, and we make it legible in the report leadership actually needs.

If leadership asked you tomorrow to prove, with data, that the people who fell last quarter no longer fall, could you do it, or could you only show how many courses were completed?

Sources and references

• Ho, G. et al., "Understanding the Efficacy of Phishing Training in Practice", 2025 IEEE Symposium on Security and Privacy. ieeexplore.ieee.org
• Lain, D., Kostiainen, K. and Čapkun, S., "Phishing in Organizations: Findings from a Large-Scale and Long-Term Study", 2022 IEEE Symposium on Security and Privacy. ieeexplore.ieee.org
• Cisco, "The 90-5-5 Concept: Your Key to Solving Human Risk in Cybersecurity", May 27, 2025. blogs.cisco.com