I favor the idea of trying to measure progress and success of the security process - Any security investment should have a quantifiable outcome in terms of risk reduction. Related to this, some months ago I read Security Metrics by Andrew Jaquith, and somewhere around the same time I read Measuring the Return on IT Security Investments authored by Intel researcher Matthew Rosenquist. The later defines a methodology for measuring the return of a security investment (ROSI) which involves the following steps,

• Evaluating cyber-attack incident data averages over time.
• Measuring the reduction of incidents from implementing new security programs.
• Valuating the impact of avoided incidents.
• Applying the results to similar areas to estimate future value.

The first step involves comparing incident counts between different months, years etc. Security is about avoiding bad things to happen, so a decrease of bad things happening must mean progress. Well, that is only true if we have an accurate value of the number of incidents. Our method of detecting incidents must be exactly as effective and accurate as last year, even though threats targeting us, vulnerabilities in our defences and assets of importance might have completely changed. If our defences (more specifically the detection process) doesn’t follow this change, we will have a inaccurate value of the number of incidents. Without measuring the accuracy of the detection process the “Number of Incidents” comparison is irrelevant. It becomes an oxomoron where we are measurably secure while being insecure.

The “number of incidents” dilemma also haunts the second step which involves measuring the reduction of incidents from implementing new security investments (technical, processes, people etc). So if we make an investment and our incident count drops then that’s considered a success, right? What if that security investment causes incidents to go undetected? Perhaps turning on IPSEC between all our servers and clients are considered a security investment - but what happened with all the alerts from IDSs that contributed to defining incidents when all traffic flyes by encrypted? Again, the model must take detection accuracy as a parameter.

The second and third step involves “valuing impact of avoided incidents” and “estimate future value” of investments. These steps aim to define how much the investments have really paid of in terms of loss avoidance. But seriously, how do you define the cost coming from an incident when it has not yet happend? Take a malware infection as an example: you can define the potential loss to be anything from the consumption of 10% cpu and memory at an individual workstation to information disclosure and theft of scetches for the next stealh aircraft. This calculation allows anyone to creatively come up with numbers that are favorable for the situation.

Measuring security progress and how security investments actually pays of is an excellent idea. But it’s difficult since both good and bad security might equal zero incidents. Somehow we must also measure how well we detect incidents and malicious activity. 

Related to yesterdays post entitled IDS and IPS systems and their effectiveness on reppelling penetration tests, Network World recently conducted a review of the prevention-ratio from of IPS:s whom are a part of UTM-products.

We tested intrusion-prevention systems in two scenarios: protecting clients (such as Web browser) and protecting servers (such as Web and e-mail servers) separately. The percentages shown are scores from our tests using Mu Security’s Mu-4000 Security Analyzer appliance to launch the attack traffic, showing what percentage of the known vulnerabilities in each category (attacks against clients, and against servers) was caught by each IPS.

The tool used in the tests are from mu Security. My, very rough, analysis of the results

• Juniper has the greatest detection and prevention rate with 87 procent catch rate of server-side attacks and 70 procent of client-side attacks. My conclusion is as such that IPS systems -at best- prevents us from roughly 80 procent of known vulnerabilities. Interestingly, this equals the ineffectiveness of anti-malware technology.
• By doing a quick overview of the results I would say that the avarage prevention rate is around 35 procent of known vulnerabilities, validating that my saying that IPS-systems in general provide very little value.

A very important thing to say about the test is that it in no way models the nature of the threat. The tests are based on publically known vulnerabilities and exploits which all of the vendors have had access to if they had care to look for them (or contact mu Security for a couple of test runs). In reality, we face issues such as evasion, noise, unkown vulnerabilities and unknown services. The results of this test are as such the best possible outcome and should be considered overly optimistic. Still interesting, and WAY better than SC Magazines attempt.

There is an ongoing debate everywere, and has been for years, about certifications, standards and regulations - in both IT/IS Security staff, related processes and procedures, management and of course security products. They are suposed to aid us in separating the wheat from the shafts - security experts from the network administrators, good software developing practices from bad, and not to say the least, good security management practices from bad.

However, I think we in generally put alot of effort in to making complex problems easy. And I think certifications is such a case. In the worst cases, I think certifications might make people, processes and companies lazy when it comes to security. Why? Because they contribute to the feeling of beeing “enough”, altough they might not be. Lets look at some examples:

What is a security expert and a professional? Many of us agree that calling ourselves experts and proffesionals might be a bad thing. We would much rather have someone else call us that, right? Here is where security certifications come in handy, because they label us as professionals and experts (like CISSP, Cerfified Information Security Proffesional), if we only pass some small tests. Most people agree that these certification tests are unrealistic, easy and doesn’t mirror the real world. So, how much expertise does actually such an expert posess? Due to the relative easy of getting such a certification, it is a short route to being aknowledge as a expert.

Similarily, what is a secure product? Common Critera and their product certifications claims to answer to that question, by evaluating products to the specificiations of “a secure product”. However, the level of detail in these specifications are low, they (can) assume unrealistic environments for the product to be placed in and as the name implies, it is all to “common”, and can be applied to all sorts of products. As with personal certifications, product certifications can be a short route to be called and considered secure. Allthough a Common Critera certification process is exhaustive and expensive, it might keep vendors from focusing on the real flaws - and once the certification is in their hand, they might consider their security efforts to be “good enough”. Which might not be the case.

In the real world, there is no guarantee that a CISSP-certified person has the level of expertise required for a security job. Similarily, a common critera certified product doesn’t mean that it won’t have flaws. In my opinion, efforts, results and experience constitutes expertise, and the security of a product isn’t conserned with facts such as “number of vulnerabilities” or “number of zero day expoits”, but how these issues, such as flaws and vulnerabilities are handled.

While studying Common Criteria evaluation-procedures, and the relationship between Protection Profiles (PP) and Security Targets (ST), I went to find out if Snort was evaluated. I recall that other opensource projects, such as SuSE 9 and some verison of Redhat, has recieved EAL4.

NIST seems to be housing information on evaluated STs and PPs, and in their intrusion dectection section I found out that SourceFire (perhaps we should call them SourcePoint, or CheckFire? ;) has their IDS sensors and management interface evaluated to EAL2, and with conformance with U.S governments PP U.S. Government Intrusion Detection System System Protection Profile, thius forfilling the requirements of IDSs protecting the whitehouse :) I believe, that this means that Snort is evaluated to EAL2 indirectly, as SourceFires sensors are relying plainly on Snort.

The summary of the evaluation says:

The evaluation was carried out in accordance with the Common Criteria Evaluation and Validation Scheme (CCEVS) process and scheme. The criteria against which the Sourcefire TOE was judged are described in the Common Criteria for Information Technology Security Evaluation, Version 2.1 and International Interpretations effective on 19, February 2003. The evaluation methodology used by the evaluation team to conduct the evaluation is the Common Methodology for Information Technology Security Evaluation, Version 1.0. Science Applications International Corporation (SAIC) determined that the evaluation assurance level (EAL) for the product is EAL 2 family of assurance requirements. The product, when configured as specified in the installation guides and user guides, satisfies all of the security functional requirements stated in the Sourcefire Intrusion Detection System Security Target. A validator on behalf of the CCEVS Validation Body monitored the evaluation carried out by SAIC. The evaluation was completed in May 2005. Results of the evaluation can be found in the Common Criteria Evaluation and Validation Scheme Validation Report for Sourcefire Intrusion Detection System, prepared by CCEVS.

Other IDS's that are certified level 2 includes ones from Cisco, Checkpoints and other big names. The product with highest EAL is IntruShield Intrusion Detection System and Symantec Manhunt Version 2.11. But this does in not mean that they are better or more secure products that the others, just that more efforts have been put into analysing them in different criterias (which none of them are how well the IDS performs in the sence of detecting attacks etc, i believe).

Interesting.