I favor the idea of trying to measure progress and success of the security process - Any security investment should have a quantifiable outcome in terms of risk reduction. Related to this, some months ago I read Security Metrics by Andrew Jaquith, and somewhere around the same time I read Measuring the Return on IT Security Investments authored by Intel researcher Matthew Rosenquist. The later defines a methodology for measuring the return of a security investment (ROSI) which involves the following steps,

• Evaluating cyber-attack incident data averages over time.
• Measuring the reduction of incidents from implementing new security programs.
• Valuating the impact of avoided incidents.
• Applying the results to similar areas to estimate future value.

The first step involves comparing incident counts between different months, years etc. Security is about avoiding bad things to happen, so a decrease of bad things happening must mean progress. Well, that is only true if we have an accurate value of the number of incidents. Our method of detecting incidents must be exactly as effective and accurate as last year, even though threats targeting us, vulnerabilities in our defences and assets of importance might have completely changed. If our defences (more specifically the detection process) doesn’t follow this change, we will have a inaccurate value of the number of incidents. Without measuring the accuracy of the detection process the “Number of Incidents” comparison is irrelevant. It becomes an oxomoron where we are measurably secure while being insecure.

The “number of incidents” dilemma also haunts the second step which involves measuring the reduction of incidents from implementing new security investments (technical, processes, people etc). So if we make an investment and our incident count drops then that’s considered a success, right? What if that security investment causes incidents to go undetected? Perhaps turning on IPSEC between all our servers and clients are considered a security investment - but what happened with all the alerts from IDSs that contributed to defining incidents when all traffic flyes by encrypted? Again, the model must take detection accuracy as a parameter.

The second and third step involves “valuing impact of avoided incidents” and “estimate future value” of investments. These steps aim to define how much the investments have really paid of in terms of loss avoidance. But seriously, how do you define the cost coming from an incident when it has not yet happend? Take a malware infection as an example: you can define the potential loss to be anything from the consumption of 10% cpu and memory at an individual workstation to information disclosure and theft of scetches for the next stealh aircraft. This calculation allows anyone to creatively come up with numbers that are favorable for the situation.

Measuring security progress and how security investments actually pays of is an excellent idea. But it’s difficult since both good and bad security might equal zero incidents. Somehow we must also measure how well we detect incidents and malicious activity. 

Last December I ranted a bit about Swedish security politics and just a couple of days ago ENISA - the European Network and Information Security Agency - released a report with recommendations on how to structure cyber security efforts within the European Union. Below are a few points that caught my attention.

Related to our lack of information on what is really going on out there:

“We recommend that the EU introduce a comprehensive security-breach notification law.”

“We recommend that the Commission (or the European Central Bank) regulate to ensure the publication of robust loss statistics for electronic crime.”

How else are we going to get a good picture of what is actually happening to others and how much it has cost them? My experience says that companies are getting really tired at hearing fictional stories and FUD. We need this.

Related to our inability of acting on issues:

“We recommend that the European Union introduce a statutory scale of damages against ISPs that do not respond promptly to requests for the removal of compromised machines, coupled with a right for users to have disconnected machines reconnected if they assume full liability.”

Internet have no boundaries. If a computer or network in Sweden is compromised and maliciously managed from Poland their respective ISP need to be responsible for disconnecting that machine and be held liable if they don’t. Our laws need teeth.

Related to vulnerabilities and patches:

“We recommend that the EU adopt a combination of early responsible vulnerability disclosure and vendor liability for unpatched software to speed the patch-development cycle.”

“We recommend security patches be offered for free, and that patches be kept separate from feature updates.”

Without a proper picture of where our vulnerabilities are, how are we supposed to be able to define our risk? Before the patch is available, how are we supposed to be able to counter a specific vulnerability with a countermeasure? A law that requires vendors to quickly notify their customers and the public about a vulnerability will allow companies to make the strategic decisions that is necessary. And of course patches should be free.

Last but not least, we need to collaborate on cybercrime.

“We recommend the establishment of an EU-wide body charged with facilitating international co-operation on cyber crime, using NATO as a model.”

Incidents happends to everyone, it’s just a question of when. Therefor, every company needs to plan for Incident Response. So does every nation and union.

Kudos to Enisa for a good report.