I favor the idea of trying to measure progress and success of the security process - Any security investment should have a quantifiable outcome in terms of risk reduction. Related to this, some months ago I read Security Metrics by Andrew Jaquith, and somewhere around the same time I read Measuring the Return on IT Security Investments authored by Intel researcher Matthew Rosenquist. The later defines a methodology for measuring the return of a security investment (ROSI) which involves the following steps,

• Evaluating cyber-attack incident data averages over time.
• Measuring the reduction of incidents from implementing new security programs.
• Valuating the impact of avoided incidents.
• Applying the results to similar areas to estimate future value.

The first step involves comparing incident counts between different months, years etc. Security is about avoiding bad things to happen, so a decrease of bad things happening must mean progress. Well, that is only true if we have an accurate value of the number of incidents. Our method of detecting incidents must be exactly as effective and accurate as last year, even though threats targeting us, vulnerabilities in our defences and assets of importance might have completely changed. If our defences (more specifically the detection process) doesn’t follow this change, we will have a inaccurate value of the number of incidents. Without measuring the accuracy of the detection process the “Number of Incidents” comparison is irrelevant. It becomes an oxomoron where we are measurably secure while being insecure.

The “number of incidents” dilemma also haunts the second step which involves measuring the reduction of incidents from implementing new security investments (technical, processes, people etc). So if we make an investment and our incident count drops then that’s considered a success, right? What if that security investment causes incidents to go undetected? Perhaps turning on IPSEC between all our servers and clients are considered a security investment - but what happened with all the alerts from IDSs that contributed to defining incidents when all traffic flyes by encrypted? Again, the model must take detection accuracy as a parameter.

The second and third step involves “valuing impact of avoided incidents” and “estimate future value” of investments. These steps aim to define how much the investments have really paid of in terms of loss avoidance. But seriously, how do you define the cost coming from an incident when it has not yet happend? Take a malware infection as an example: you can define the potential loss to be anything from the consumption of 10% cpu and memory at an individual workstation to information disclosure and theft of scetches for the next stealh aircraft. This calculation allows anyone to creatively come up with numbers that are favorable for the situation.

Measuring security progress and how security investments actually pays of is an excellent idea. But it’s difficult since both good and bad security might equal zero incidents. Somehow we must also measure how well we detect incidents and malicious activity. 

In response to a post on risk versus uncertainty.

“people spend too much time trying to reduce uncertainty and too little time focusing on reducing risk.”

My impression is the exact opposite. Companies spend to much time (and money!) on ad-hoc attempts in reducing risk with no control of where their biggest risks are or how these countermeasures actually pays of in terms of risk reduction. There is too much focus on headline-threats and efforts resembling “fire-fighting” and “socker-goal security”. Companies buy firewalls, intrusion prevention systems, data leak prevention solutions for millions so the can put them into place and forget them. There is too much uncertainty in daily security operations, which is why I think that reducing uncertainty is crucial. Companies often can’t answer the simplest questions. I say implement solutions that give you insights in vulnerabilities, threats, assets and ultimately risks (no, the answer isn’t an annual risk-analysis paper exercise). Then (!) implement measures for risk reduction.

Apperently, the growth of the number of disclosed vulnerabilities are declining. As usuall, these kind of statistics follows with alot of speculations of what this means for security. A common idea is that fewer disclosed vulnerabilities must mean we are more secure, right?

While at it, let me speculate some too. From a hackers perspective, the most valuable vulnerability is the one no-one else knows, right? Such a vulnerability, if in the “right” (from our perspective, wrong) piece of code, would let a hacker walk in through the front door of many targets. With money to gain from stealing company secrets, identity information and credit card details, it’s relatively obvious that vulnerabilities themselves have more value today than earlier. The decline in disclosed vulnerabilities could just as easily be a result of an increased care, focus and determination within the underground. For use in targeted, more sophisticated attacks. There is no easy way for us to know exactly, and that is why drawing conclusions of the state of security from statistics of vulnerabilities is far to simplistic.

This kind of mindset reminds me of a speech I heard from the CSO of a large financial company here in Sweden: “We measure our security on the number of security incidents we have a year”. I instantly thought, “That says nothing. How hard are you looking?”.