I favor the idea of trying to measure progress and success of the security process - Any security investment should have a quantifiable outcome in terms of risk reduction. Related to this, some months ago I read Security Metrics by Andrew Jaquith, and somewhere around the same time I read Measuring the Return on IT Security Investments authored by Intel researcher Matthew Rosenquist. The later defines a methodology for measuring the return of a security investment (ROSI) which involves the following steps,

• Evaluating cyber-attack incident data averages over time.
• Measuring the reduction of incidents from implementing new security programs.
• Valuating the impact of avoided incidents.
• Applying the results to similar areas to estimate future value.

The first step involves comparing incident counts between different months, years etc. Security is about avoiding bad things to happen, so a decrease of bad things happening must mean progress. Well, that is only true if we have an accurate value of the number of incidents. Our method of detecting incidents must be exactly as effective and accurate as last year, even though threats targeting us, vulnerabilities in our defences and assets of importance might have completely changed. If our defences (more specifically the detection process) doesn’t follow this change, we will have a inaccurate value of the number of incidents. Without measuring the accuracy of the detection process the “Number of Incidents” comparison is irrelevant. It becomes an oxomoron where we are measurably secure while being insecure.

The “number of incidents” dilemma also haunts the second step which involves measuring the reduction of incidents from implementing new security investments (technical, processes, people etc). So if we make an investment and our incident count drops then that’s considered a success, right? What if that security investment causes incidents to go undetected? Perhaps turning on IPSEC between all our servers and clients are considered a security investment - but what happened with all the alerts from IDSs that contributed to defining incidents when all traffic flyes by encrypted? Again, the model must take detection accuracy as a parameter.

The second and third step involves “valuing impact of avoided incidents” and “estimate future value” of investments. These steps aim to define how much the investments have really paid of in terms of loss avoidance. But seriously, how do you define the cost coming from an incident when it has not yet happend? Take a malware infection as an example: you can define the potential loss to be anything from the consumption of 10% cpu and memory at an individual workstation to information disclosure and theft of scetches for the next stealh aircraft. This calculation allows anyone to creatively come up with numbers that are favorable for the situation.

Measuring security progress and how security investments actually pays of is an excellent idea. But it’s difficult since both good and bad security might equal zero incidents. Somehow we must also measure how well we detect incidents and malicious activity. 

Apperently, the growth of the number of disclosed vulnerabilities are declining. As usuall, these kind of statistics follows with alot of speculations of what this means for security. A common idea is that fewer disclosed vulnerabilities must mean we are more secure, right?

While at it, let me speculate some too. From a hackers perspective, the most valuable vulnerability is the one no-one else knows, right? Such a vulnerability, if in the “right” (from our perspective, wrong) piece of code, would let a hacker walk in through the front door of many targets. With money to gain from stealing company secrets, identity information and credit card details, it’s relatively obvious that vulnerabilities themselves have more value today than earlier. The decline in disclosed vulnerabilities could just as easily be a result of an increased care, focus and determination within the underground. For use in targeted, more sophisticated attacks. There is no easy way for us to know exactly, and that is why drawing conclusions of the state of security from statistics of vulnerabilities is far to simplistic.

This kind of mindset reminds me of a speech I heard from the CSO of a large financial company here in Sweden: “We measure our security on the number of security incidents we have a year”. I instantly thought, “That says nothing. How hard are you looking?”.

There is an ongoing debate everywere, and has been for years, about certifications, standards and regulations - in both IT/IS Security staff, related processes and procedures, management and of course security products. They are suposed to aid us in separating the wheat from the shafts - security experts from the network administrators, good software developing practices from bad, and not to say the least, good security management practices from bad.

However, I think we in generally put alot of effort in to making complex problems easy. And I think certifications is such a case. In the worst cases, I think certifications might make people, processes and companies lazy when it comes to security. Why? Because they contribute to the feeling of beeing “enough”, altough they might not be. Lets look at some examples:

What is a security expert and a professional? Many of us agree that calling ourselves experts and proffesionals might be a bad thing. We would much rather have someone else call us that, right? Here is where security certifications come in handy, because they label us as professionals and experts (like CISSP, Cerfified Information Security Proffesional), if we only pass some small tests. Most people agree that these certification tests are unrealistic, easy and doesn’t mirror the real world. So, how much expertise does actually such an expert posess? Due to the relative easy of getting such a certification, it is a short route to being aknowledge as a expert.

Similarily, what is a secure product? Common Critera and their product certifications claims to answer to that question, by evaluating products to the specificiations of “a secure product”. However, the level of detail in these specifications are low, they (can) assume unrealistic environments for the product to be placed in and as the name implies, it is all to “common”, and can be applied to all sorts of products. As with personal certifications, product certifications can be a short route to be called and considered secure. Allthough a Common Critera certification process is exhaustive and expensive, it might keep vendors from focusing on the real flaws - and once the certification is in their hand, they might consider their security efforts to be “good enough”. Which might not be the case.

In the real world, there is no guarantee that a CISSP-certified person has the level of expertise required for a security job. Similarily, a common critera certified product doesn’t mean that it won’t have flaws. In my opinion, efforts, results and experience constitutes expertise, and the security of a product isn’t conserned with facts such as “number of vulnerabilities” or “number of zero day expoits”, but how these issues, such as flaws and vulnerabilities are handled.

For a engineer like me, numbers are like play-doh. But as a security enthusiast, I’m not spoiled with figures on the costs of attacks, exploits and similar. But here are some of the official cases that I’ve come across..

During December, a zero day exploit for the WMF-vulnerability was sold by russian hackers for roughly $4000 a piece

Also in December, a zero day exploit for an undisclosed vulnerability in Microsoft Excel was put for auction at Ebay - the bid reached $59, and had the time to attract 19 offers until it was removed by administrators.

Details of a single credit card is worth roughly $1 (0.83 euros) and details of a a card with a three digit pin for $5. Additional personal information such as social security numbers and similar might add another $100. Accounts with high balance might go as far as $100. I’ve also read somewhere that a single screendump from a banks teller’s terminal, showing account holder information and related facts, is worth about $400

iDefense, a lim of Verisign who performs vulnerability-based services, offers $10000 for un-disclosed vulnerabilities in Microsoft Windows.

Mozilla offers a $500 bug bounty for undisclosed critical flaws.

Microsoft offered $250000 for information leading to the arrest of the Sasser-author.

For $15 you could buy a virus creation kit, from Russia.