Measure security awareness effectiveness: an MSP KPI guide

TL;DR

To measure security awareness effectiveness, MSPs need more than a green completion dashboard. Completion tells you whether training was delivered. It does not tell you whether users report suspicious messages, whether repeat risk is falling, or whether the client has evidence it can use during QBRs, audits, and insurance renewals.

A better measurement stack has 4 layers: coverage, behavior, risk movement, and client-ready evidence. Track who is covered, who completed, who reported, who needs help, which exceptions remain open, and whether the report can be explained to a client leader without a spreadsheet tour.

What does it mean to measure security awareness effectiveness?

Measuring security awareness effectiveness means asking a blunt question: did the program change what people do when they meet risk?

That is different from asking whether the training platform sent a module, whether a certificate exists, or whether a quiz was passed. Those are useful records. They are not the whole answer.

NISTIR 8420A describes cybersecurity awareness programs as efforts that help people recognize and respond to security issues, with the goal of long-term behavior change. It also notes a common challenge: teams struggle to measure program impact, and staff often see training as a boring check-the-box activity. That is the trap MSPs need to avoid for clients.

For an MSP, effectiveness has another layer. The client does not only need internal security improvement. They also need a report that proves what was done, what changed, and what still needs attention. If the MSP cannot explain the metrics in a client QBR, audit pack, or insurance renewal conversation, the numbers are not yet service-ready.

So the working definition is simple:

Security awareness is effective when more people are covered, more people act safely, risky behavior trends down, real suspicious activity is reported faster, and the client can see the evidence.

Why this matters for MSPs

MSPs deliver security awareness across messy client environments. Some clients have clean identity data. Some have stale users. Some have seasonal staff, shared mailboxes, privileged users, executives, contractors, and groups that never quite match the billing list.

That makes measurement harder than it looks.

A single client might ask, “Are we compliant?” The MSP has to translate that into a set of narrower questions:

• Who was in scope?
• Was training assigned to the right people?
• Who completed it?
• Who ignored it?
• Which roles need deeper training?
• Did users report suspicious messages?
• Did repeat risky actions improve?
• What evidence can be shown to an auditor, insurer, or board?

CIS Control 14 is useful here because it frames security awareness as a maintained program that influences workforce behavior and reduces cyber risk. Its assessment language points MSPs toward records that can be measured: workforce lists, completion dates, training currency, content review, and role-relevant training.

CISA’s public phishing guidance adds the operational side. Users should learn to recognize and report phishing, not just avoid clicking. A user who reports a convincing phish quickly is doing something useful for the client. A dashboard that only measures “did not click” can miss that.

The business reason is just as important. If reporting depends on manual exports, one-off spreadsheets, and client-by-client clean-up, security awareness becomes another margin drain. Measurement needs to become a repeatable MSP service workflow, not an audit-week scramble.

The KPI stack MSPs actually need

Do not start with 25 metrics. Start with enough to explain the client’s position.

A useful measurement stack has 4 layers.

Layer	KPI	What it answers	Why MSPs need it
Coverage	In-scope users, assigned users, active users excluded	Are the right people covered?	Finds gaps before the client thinks training is universal
Completion	Completion rate, overdue users, completion age	Did training happen on time?	Supports compliance and follow-up workflows
Behavior	Phishing report rate, time-to-report, repeat risky actions, real-threat reports	Are users acting differently?	Shows movement beyond certificate collection
Risk context	High-risk-role coverage, privileged-user training, executive participation	Are the riskiest groups included?	Stops average scores hiding important gaps
Evidence	Report generation date, source records, exceptions, client-ready summary	Can the client use the proof?	Turns platform data into QBR, audit, and insurance material

The table matters because each metric has a job. Completion is not bad. It is just a coverage and evidence metric. Treating it as the full measure of effectiveness creates false confidence.

Step-by-step: how to measure security awareness effectiveness

1. Lock the population before looking at results

Start with scope. Effectiveness metrics are only meaningful when the denominator is honest.

For each client, define the active population for the reporting period. Include the date range, tenant, user source, exclusions, and special groups. New starters, leavers, contractors, privileged users, shared accounts, and service accounts should not be mixed without notes.

This is where many reports go wrong. A client sees “92% complete” and assumes 92% of the company is covered. If the source list excluded 40 casual staff or 12 executives, the metric is weak.

The first MSP question should be: “92% of whom?”

2. Separate completion from effectiveness

Completion rate is useful because it shows whether the basic program reached people. It is also useful for compliance evidence. CIS Control 14 assessment material looks at whether workforce members completed training and whether it is current.

But completion rate does not show whether people behave differently.

Use completion for these questions:

• Were users assigned training?
• Did they complete it within the required period?
• Which users are overdue?
• Which groups need manager follow-up?
• Is the training current, or is it older than the accepted refresh period?

Do not use completion for these claims:

• “The client is secure.”
• “Human risk is reduced.”
• “Users can spot phishing.”
• “The client is compliant with the whole framework.”

Those claims need wider evidence.

3. Measure reporting, not only failure

Phishing simulations often lead teams to obsess over click rate. Click rate has a place, but it is easy to misread. Harder lures usually produce higher clicks. Easier lures produce lower clicks. Without controlling for difficulty, click rate can punish a more realistic test and reward a weak one.

NIST’s Phish Scale work matters because it treats phishing difficulty as something that can be assessed, not ignored. If one quarter uses a blatant fake invoice and the next uses a realistic payroll lure, the results should not be compared as if the test stayed constant.

Report rate is a stronger behavior signal. It measures whether users notice something suspicious and raise it through the right channel. Time-to-report adds urgency: the first useful report can help an MSP or security team act before more users are exposed.

Track:

• report rate for simulations,
• report rate for real suspicious messages where available,
• first report time,
• median report time,
• report quality,
• and whether reporting improves after training.

A user who reports a real phish is not a passive learner. They are part of the detection layer.

4. Track repeat risk without turning it into a shame list

Every program has repeat risky actions. Someone clicks multiple simulations. Someone ignores every reminder. Someone reports only after opening the attachment. Someone in finance keeps interacting with invoice lures.

Those patterns matter, but the goal is improvement, not humiliation.

A useful repeat-risk view groups users or roles by trend:

• repeat risky action improved,
• repeat risky action unchanged,
• repeat risky action getting worse,
• no recent test or signal,
• needs manager or MSP follow-up.

For client reporting, avoid turning this into a public name-and-shame table. Give the client enough detail to act. Keep individual-level records where needed for follow-up, role-based coaching, or audit evidence.

This is also where high-risk roles deserve attention. Executives, finance, HR, IT admins, and helpdesk staff often need deeper or more specific training than a generic annual module. CIS Control 14 calls out different risks by role. MSP reports should too.

5. Measure evidence quality

Security awareness metrics should be useful outside the training platform.

A client QBR needs a clean summary. An ISO 27001 evidence request needs scope, dates, and source records. A cyber insurance questionnaire needs precise answers, not a raw export. A board report needs trend and decision points.

Evidence quality is a metric in its own right. Ask:

• Can the report show the client tenant only?
• Does it show the reporting period?
• Does it show the user population and exclusions?
• Does it show assignments, completions, dates, and overdue users?
• Does it explain exceptions?
• Does it separate awareness evidence from other controls?
• Can the MSP reproduce the report later?

If the answer is no, the program may be running, but the managed service is not yet mature.

6. Build a monthly and quarterly rhythm

Security awareness effectiveness should not be checked only when a client asks for proof.

Use 2 cadences.

Monthly operational review:

• new users added,
• overdue users,
• campaigns assigned,
• reminders sent,
• report rate and first-report time,
• repeat-risk follow-up,
• exceptions needing client action.

Quarterly client review:

• coverage trend,
• completion trend,
• behavior trend,
• high-risk-role view,
• major risks observed,
• evidence pack status,
• next-quarter training focus.

The monthly review keeps the service working. The quarterly review helps the client understand value.

What good looks like

Good measurement is boring in the right places and sharp where it matters.

A strong MSP security awareness report has these traits.

First, the scope is obvious. The report names the client, tenant, reporting period, and user population.

Second, the metrics are layered. Completion sits beside reporting, behavior, high-risk-role coverage, and exceptions.

Third, the report separates activity from outcome. It does not pretend that a completed module proves behavior change.

Fourth, the report is client-readable. A business owner can understand the point without opening a CSV.

Fifth, the report creates a next action. The client can approve reminders, ask managers to follow up, add a high-risk group, or accept a documented exception.

A good report might say:

• “Training coverage increased from 84% to 96% after user-list cleanup.”
• “17 users are overdue; 11 are new starters inside the agreed grace period.”
• “Phishing report rate improved, but finance had 3 repeat risky actions.”
• “Executives completed baseline training, but have not yet completed role-specific BEC training.”
• “Q3 action: add finance and leadership modules, keep monthly exception review.”

That is useful. It gives the MSP and client something to do.

Mistakes to avoid

Treating completion as the whole story

Completion proves activity. It does not prove judgment under pressure.

Comparing phishing click rates without difficulty context

A harder simulation can make the program look worse even if the test is better. Use difficulty notes and pair click rate with reporting and repeat-risk metrics.

Reporting blended fleet data to a client

Fleet dashboards help the MSP manage service delivery. Client reports need tenant-specific evidence.

Hiding exceptions

Exceptions do not make a report weak. Hidden exceptions do. New starters, leavers, inactive accounts, and client-approved exclusions should be shown clearly.

Overclaiming framework coverage

Security awareness evidence can support NIST, CIS, ISO 27001, Essential Eight, and insurance conversations. It does not prove the whole framework is implemented.

Using metrics with no owner

Every metric should have an owner and a next action. If 19 users are overdue, who follows up? If report rate is low, what changes next month? If executives are excluded, who fixes scope?

Framework mapping: what the evidence can support

Security awareness effectiveness metrics often help with compliance and risk conversations, but they need careful labels.

NISTIR 8420A is useful for program design because it links awareness to long-term behavior change and highlights the measurement challenge. NIST’s Phish Scale work is useful for simulation interpretation because phish difficulty affects user behavior.

CIS Control 14 is useful for operational evidence. It looks at a maintained program, training at hire and at least annually, content review, completion dates, and role-relevant topics.

CISA’s phishing guidance is useful for user behavior. It reinforces that people need to recognize and report phishing, which supports report-rate and time-to-report metrics.

FBI IC3 data gives context for why email trust abuse still matters. The 2025 IC3 report lists phishing/spoofing as the highest complaint-count crime type, and business email compromise among the largest reported loss categories. Use that as risk context, not as proof that any one client’s training works.

A safe mapping looks like this:

Evidence type	Supports	Should not claim
Completion roster	Training coverage and current completion	Behavior change by itself
Phishing report rate	Users are recognizing and escalating suspicious messages	Full phishing resilience
Time-to-report	Speed of user escalation	Incident containment by itself
Repeat-risk trend	Whether coaching and nudges are changing risky actions	Guaranteed breach reduction
High-risk-role coverage	Whether executives, finance, HR, and admins are included	That role-specific threats are solved
Client evidence pack	Audit, insurance, and QBR reporting support	Whole-framework compliance

How a flat-rate MSP SAT platform helps

Measurement gets harder when coverage is rationed by seat count.

If every new learner adds cost, MSPs can be tempted to narrow scope, delay onboarding, or treat training as an add-on. That weakens the denominator. It also makes reporting harder because the MSP has to explain who was not included and why.

Defendwise is built around a different MSP model: $399/month flat, unlimited users, unlimited client organisations, multi-tenant management, white-label delivery, automated onboarding, reminders, and reporting. That does not remove the need for MSP review. It does make broad coverage, tenant-separated reporting, and client-ready evidence easier to package without per-seat margin anxiety.

The platform should not be the hero of the report. The client outcome should be. More users covered. Less admin drag. Better evidence. Clearer QBR conversations.

If you want to connect this to the wider MSP operating model, start with the Defendwise blog, the coverage gap guide, the hidden admin cost guide, and DefendWise vs KnowBe4 for MSPs.

Frequently asked questions

How do you measure security awareness effectiveness?

Use a small KPI stack that covers activity, behavior, risk context, and evidence. At minimum, track in-scope users, training assignment, completion, overdue users, phishing report rate, time-to-report, repeat risky actions, high-risk-role coverage, and exceptions.

Is completion rate enough to prove security awareness training works?

No. Completion rate is useful for coverage and compliance evidence, but it does not prove users can spot or report a real attack. Pair completion with behavior metrics.

What is a good security awareness KPI for MSP client QBRs?

For QBRs, use metrics the client can act on: coverage trend, overdue users, report rate, repeat-risk trend, high-risk-role coverage, and open exceptions. Avoid dumping platform-only metrics with no decision attached.

Should MSPs report phishing click rate?

Yes, but not alone. Phishing click rate depends on simulation difficulty, user population, timing, and lure design. Use it with report rate, time-to-report, difficulty notes, and repeat-risk improvement.

How often should security awareness metrics be reviewed?

Review operational metrics monthly and client-facing trend metrics quarterly. Produce separate evidence packs for audits, insurance renewals, or incident-related requests.

What should an MSP do when a client has low report rate?

First, check whether users know how to report. Then simplify the reporting path, reinforce it in training, test realistic lures, and praise useful reports. Low report rate is often a workflow problem before it is a user problem.

Can security awareness metrics prove compliance?

They can support compliance evidence, especially for awareness and training controls. They should not be labelled as proof of full ISO 27001, NIST, CIS, Essential Eight, or cyber insurance compliance unless the wider evidence exists.

How does Defendwise help MSPs measure security awareness effectiveness?

Defendwise supports the MSP workflow around broad coverage, multi-tenant management, white-label delivery, automated onboarding, reminders, and reporting. MSPs still need to review scope, exceptions, and claims before using the output in client reporting.

Sources

• NISTIR 8420A, “Approaches and Challenges of Federal Cybersecurity Awareness Programs”: https://nvlpubs.nist.gov/nistpubs/ir/2022/NIST.IR.8420A.pdf
• NIST, “Phishing With a Net: The NIST Phish Scale and Cybersecurity Awareness”: https://www.nist.gov/publications/phishing-net-nist-phish-scale-and-cybersecurity-awareness
• CISA, “Recognize and Report Phishing”: https://www.cisa.gov/secure-our-world/recognize-and-report-phishing
• CISA, “Cybersecurity Awareness & Training”: https://www.cisa.gov/cybersecurity-awareness-training
• CIS Controls Assessment Specification, Control 14: https://cas.docs.cisecurity.org/en/latest/source/Controls14
• CIS, “Security Awareness and Skills Training”: https://www.cisecurity.org/controls/security-awareness-and-skills-training
• FBI IC3, 2025 Annual Report: https://www.ic3.gov/AnnualReport/Reports/2025_IC3Report.pdf
• Verizon, Data Breach Investigations Report archive and 2025 report link: https://www.verizon.com/business/resources/reports/dbir/
• Brightside AI, “How to Measure Security Awareness Training Effectiveness”: https://www.brside.com/blog/how-to-measure-security-awareness-training-effectiveness
• CybSafe, “Measuring the effectiveness of security awareness programs”: https://www.cybsafe.com/blog/measuring-the-effectiveness-of-security-awareness-training
• Hoxhunt, “Security Awareness Training: Examples, Metrics & Frameworks”: https://hoxhunt.com/guide/security-awareness-training
• Proofpoint, “Security Awareness Training Effectiveness”: https://www.proofpoint.com/us/blog/security-awareness-training/security-awareness-training-effectiveness