Our analog world is full of vulnerabilities. I enjoy watching shows like Tiger Team and The Real Hustle because they offer good examples of their great numbers, and how easily they can be exploited. Interestingely enough, we don’t put alot of effort into minimising them either. Instead we accept risks, or transfere them with for instance insurance. We are more focused on detection and response rather than prevention. Likely because it’s evolved as the most cost effective method.

I believe paramount to why this works in the analog world, is because incidents are naturally detectable. If someone steals your juwelery you’ll eventually miss it. We know how much the loss cost, and since all of us have insurance we (or at least the insurance companies) have good averages on what a normal person is likely to loose per year and bases their premiums on that.

But a digital asset can be stolen in a number of ways without the owner knowing about it. Digital incidents aren’t naturally detectable, and we have no real numbers on the number of incidents and the average costs associated with them. In fact, there are insurances for digital security breaches and issues, but if we fail to detect them, when are we ever going to make us of our insurance? It will never be the most cost effective method. It seems like we have no option other than employing prevention and chasing vulnerabilities. But then again, since there are no perfect security, this approach will always leave us with an inaccurate view of incidents and no other options. Just another hamster wheel of pain. Find vulnerability, Patch and Proceed.

Below is a brief taxonomy of various events that I like to use when thinking about security monitoring. Their naming can be questioned, but I believe they give a decent structure to various events and their relationships. It like to think that it aligns well with NSM.

Security Events

Security events are any form of warnings, or what I like to treat them as, “indications” of attacks and/or malicious activities. From a security monitoring perspective, these events can direct the analysis and response activities in a direction where it is likely that something security relevant is happening. Security events can be an IDS/IPS alert from triggered signatures indicating that someone is targeting exploits at our public web services, or a network traffic anomaly alert indicating that an unusual high amount of traffic is leaving our network in the middle of the night. Security events need not be limited to digital ones, but can also be a user calling the help desk saying that his/her computer is slow.

Typical information sources are network/host intrusion detection/prevention systems, web proxies with malware features, endpoint antivirus systems etc. Any system that “looks” for attacks.

Communication Events

Communication events are anything that tell us who is talking to who. Collecting these events allows for establishing, depending on the type events of course, granular “audit trails”. A complete picture of how a certain host has been communicating can assist the response process by limiting the scope of a possible compromise, or to completely dismiss alerts in the case no traffic or data has been exchanged between the target and the source.

Typical information sources for informing us about communication patterns are Firewalls in the forms of Accept and Deny events; and Routing and Switching infrastructure by exporting meta data about sessions and traffic in the form of traffic flow information (i.e. netflow, sflow etc). Session data and network traffic flow information is favourable before firewalls. The ultimate information source for Communication Events are systems dedicated to sampling of full content data from traffic at strategic positions in the network. Full content data offers complete transaction records of sampled traffic, and can often completely verify or dismiss a suspected attack (unless it’s encrypted), but at the cost of alot of diskspace.

Identity Events

Identity events are any form of event that ties an IP-adress to another type of identity, i.e. a user name, a mac adress, a dns-name, a physical location such as a room-numer, or even a telephone number or in the case of an external host complete whois and owner/abuse information. Identitity events allows for establishing relationships between different identities and assets. A relationsship between an IP-adress and a Username aids the analysis and response process by making it possible to ask questions to the person in question.

Typical information source for Identity events are ordinary clients and servers, central authentication systems, dhcp server and dns server.

Activity Events

Activity events are information about activities, and not necessarily strictly security related ones. If communication events informs about who is talking to who, activity events informs about what they are saying and what they are talking about. Activity events allows us to know what is happening in a given situation/session/point in time - i.e. what actions hosts, users or processes are making. Having a complete picture of the HTTP requests a client have been making towards the internet can support the reponse of a possible malware infection of a client. Having a complete picture of what the IIS- and SQL server are up to eases the investigation of a possible XSS or SQL-injection attack.

Typical informations sources for Activity events are any strategic system that has a certain service. Web Proxies tracks any web requests to the internet. A web server, or a web application firewall, tracks interactions with the web server. Collecting Event logs from a Windows server or syslog feeds from an important Solaris servers allow you to track interactions with those resources.

To summarise:

Security Events point us to likely malicious activity. Communication events tells us who is talking to who. Identity Events tells us who the source and target is and Activity events tells us what actions a system, a users or a process is doing. My experience tells me that a successfull security monitoring requires a degree of all four of these.

I’ve spent alot of time lately writing about how various security technologies fail. My idea is that the problem boils down to the fact that most of our security efforts are vulnerability oriented. Fixing the vulnerabilities seem to be the smartest thing to do, right? Problems is, we can’t keep up with patching and we aren’t aware of all our vulnerabilities, or even a small part of them. To some degree, we will always be vulnerable. In order to complement the natural patch management process I favor efforts and technologies that are threat centric instead of vulnerability centric; Meaning efforts that focus on preventing threats instead of instances of attacks at vulnerabilities.

In an attempt to make an example, have a look at authentication systems. Most of us understand that all passwords can be broken if the necessary time and resources are put into it. The security of the authentication system relies on the strength of passwords, and in order to higher the bar for brute forcing it we

• create a basic policy that defines the minimum strength of the password (eight chars, with numbers etc).
• we force the user to change the password every four, six or eight weeks.
• we only let the user make three unsuccessfull attempts before disallowing any further attempts (or we incrementaly increase the delay between each possible attempt).

The first two points are vulnerability oriented, meaning they attempt to increase the strength of the password and as such decrease the vulnerability of the password. They increase the level of effort the threat must put into brute forcing the password. The last one, however, focuses on preventing any attempts of guessing the password. It doesn’t increase the strength of the password, but drastically limits the threats possibility to repeatably try passwords. This measure is threat oriented and no matter how much money or time he or she puts into buying equipment for crunching numbers, all they have is three attempts (there are other ways of defeating authentication systems though).

This kind of threat centric approach is what most Intrusion Prevention technologies lack. Common IPS:s functions as authentication systems without the threat centric part; They block individual attacks but passively lets the intruder keep trying until he crafts an exploit that slips through. Good examples are solutions that correlates alerts with vulnerability information of the target (which ISS does) in order to decide what to block. They forget to question whether the activity itself is acceptable. Just because the webserver runs Apache, all attempts at exploiting it as an IIS is ok? Or just because a perticular vulnerability isn’t present in this version of Apache, the activity is ok? This kind of thinking is, for example, what causes IPS implementation to fail at repelling penetration tests.

To be fair, some IPS:s have the necessary functionallity for being a threat centric tool. Juniper, for instance, has the option to block individual offending packets, an offending session and ultimately an offending IP for a configurable amount of time. Other IPS:s has similar functionality. Problem is that the functionallity isn’t used in an automated fashion due to the risk of consequences from creating long lasting blocks of legitimate traffic sources. This is why I advocate the involvement of a human to make the ultimate decision to block an offending IP for a given time, or use a good SIEM system (Security Information and Event Management) that correlates alerts with other information sources to get the knowledge of context necessary to make accurate decisions.

Expect to see more posts related to threat centric security in the future.

Related to yesterdays post entitled IDS and IPS systems and their effectiveness on reppelling penetration tests, Network World recently conducted a review of the prevention-ratio from of IPS:s whom are a part of UTM-products.

We tested intrusion-prevention systems in two scenarios: protecting clients (such as Web browser) and protecting servers (such as Web and e-mail servers) separately. The percentages shown are scores from our tests using Mu Security’s Mu-4000 Security Analyzer appliance to launch the attack traffic, showing what percentage of the known vulnerabilities in each category (attacks against clients, and against servers) was caught by each IPS.

The tool used in the tests are from mu Security. My, very rough, analysis of the results

• Juniper has the greatest detection and prevention rate with 87 procent catch rate of server-side attacks and 70 procent of client-side attacks. My conclusion is as such that IPS systems -at best- prevents us from roughly 80 procent of known vulnerabilities. Interestingly, this equals the ineffectiveness of anti-malware technology.
• By doing a quick overview of the results I would say that the avarage prevention rate is around 35 procent of known vulnerabilities, validating that my saying that IPS-systems in general provide very little value.

A very important thing to say about the test is that it in no way models the nature of the threat. The tests are based on publically known vulnerabilities and exploits which all of the vendors have had access to if they had care to look for them (or contact mu Security for a couple of test runs). In reality, we face issues such as evasion, noise, unkown vulnerabilities and unknown services. The results of this test are as such the best possible outcome and should be considered overly optimistic. Still interesting, and WAY better than SC Magazines attempt.

I argue that IDS and IPS system by default are of little use as protection from targeted, focused, penetration attempts. While some of these systems provides the means of doing so, the lack processes, routines, a human mind, eyes and actions cripples them.

Here’s an interview that highlights these systems inability to do just that. From the interview,

In the majority of cases, they [read: IDS/IPS] just don’t end up doing what they were purchased for. An easy test that most fail is with basic port scans (that almost all are configured to pick up). We assume most are picking up “loud� scans (really fast and obvious scans with no attempt to be sneaky about what we are doing), but few people are pulling us up on this. (Keep in mind, with a majority of our tests, we recommend that clients don’t tell the operations team responsible for monitoring these devices that we are going to test – thereby, we also test the response effectiveness).

Validation and feedback is one of the pillars of security. It’s the only way that these solutions provide security assurance - they don’t do that right out of the box.

This is also spot on

An IPS only forces the attacker to know their exploits better, and take things slower. For instance, an IPS may drop all packets that have NOP sleds in them (0×909090 etc) which is used in a lot of (kind of sloppy) buffer overflows. It is however possible for an attacker to stop the IPS from seeing this.

What fails here, as well as in the earlier port-scan example, is the lack of response. Any alert or action that comes from a targeted penetration attempt should be followed by watchlisting relevant IPs and looking for further signs and attempts, possibly looking at capturing flow and full content data and other types of logs etc. Blocking shouldn’t be limited to the specific session that holds the exploit, but more importantly any following traffic. That will make the assessor work for their money.

This is a good piece from Microsoft where they mention the value of security monitoring. From the article

The primary goal of a security monitoring and attack detection system is to help identify suspicious events on a network that may indicate malicious activity or procedural errors.

Microsoft apperently understands the importance of really looking for indications of attacks, instead of living in belief that systems are impenetrable and that security products will offer protection. Their suggestions with regards to the actual process and routine is also accurate:

A security monitoring solution is actually a continual process of planning, implementing, managing, and testing, because that is the very nature of security monitoring. Because the threats to business networks are always changing, the system that monitors the security in a business network must also change.

This process is suggested as a part of their Microsoft Operations Framework (MOF), which also features other operational routines.

I like that Microsoft is putting some effort in the defining the operational requirements for managing their systems and environments. The lack of these kind of security operations are what makes companies fail, not that they don’t have enough security products. The article also features some good hints on analysing Window Event Logs.

A post at the Snort-users mailing list took me to the “best intrusion detection/prevention solution”-competion at SC Magazine. First, is it actually a competition? Deciding something by voting is what I call an election?

Leaving that aside, two things strikes me.

First, what are the critera for deciding what is best? Best as protection against security incidents, most good looking, easiest to administer or the least pricey? Have the voters something to compare with? The results would most likely point to “the most used”, not “the best in terms of protection”. When the results of the “competition” is presented and Snort is the winner, then I consider myself right :)

Secondly, how have the nominees been decided? Among the nominees I see a product that only does wireless (AirDefence), a product that only looks at the host (CA HIPS), a UTM-product (Fortinet), a NBAD-product (lancope), and finally, but also the most striking: an IPS as a managed service (Verisign). Hell, on what criteria have they decided upon these vendors/products? And where are their competitiors such as Juniper, TopLayer, and why not Bro, and where are all the SIEM-solutions?

NSS has the closest thing to a good evaluation standard of IDS/IPS-system. Not perfect though, but I hope people look, and pay, for those reports instead of making decisions on the results of crappy “competitions”.

What many people seem to forget when thinking about IDS and especially IPS-systems is the important tie they have to performing network forensics. Not all attacks will be blocked, and the ones that are detected by other means needs to be investigated. What equipment is better suited for that than a device that sits inline, at the perimiter or core, monitors all sessions entering and leaving the premises, and that at least tries to identify possibly malicious elements of the monitored traffic. That’s right, none. The true value of a modern monitoring system is that they are a tool for investigating security incidents - not just blocking them.

Having this in mind, this puts new requirements on the IDS/IPS. Not only should they have a good detection rate and low false positive rate, they should also have the necessary information available to actually investigate the indications that alerts truly are, but also investigate traffic that isn’t tied to an alert. Further more, which is one of the crucial elements of performing forensics, is to avoid informing the potential attacker about your knowledge of an attack. This is lesson 1A in incident handling - never try to contact the source of the alerts; never visit the website that the attack is coming from etc. All these actions might inform the attacker of your knowledge of the compromise, and as such will cripple your ability to investigate it.

Related to this is the fact that most IDS/IPS systems, and some firewalls aswell for that matter, does lookups of source and target IPs in the alerts. This feature, which is intended to make the alerts more easily readable, might in fact inform the attacker that some system has processed the traffic from him. When an alert fires the system will automatically try to do a reverse lookup on the IP, the host will contact the root server and follow it’s trail down to the DNS server that has information about the IP in question. This DNS-server could very likely be run by the attacker who now sees someone doing lookups on his IP. See how the solution itself contributes to the problem?

A possible solution to this, which doesn’t involve turning of name resolution entirely, is to be able to configure the system to use 3:rd party dns-servers or dns-proxies for external IP:s. I learnt of a tool that has implemented the support for this, namely the excellent network security monitoring console Sguil.

A common question when discussing intrusion detection versus intrusion prevention is “why on earth would we want to do only detection?”. The pretty simple reason is (which I have written about tons of times) the inaccuracy of finding and identifying something that is a very small part of the entire set looked at. I think we can learn alot in performing security in the digital world from the analog. For instance, why is it that we have very few systems in the analog world that claims to prevent something once detected? The security cameras in the subway isn’t hooked up with stun guns right? The “X-ray station” at the airport is monitored by real personel, why is that? And why not having a way of automatically stopping a car that is violating the traffic control signs? Because it doesn’t work. Why would the digital world be any different?

I learnt from Schneier of a good example of the difficulty of performing monitoring and incident detection, in the real world. From the article,

The 178 video cameras that keep watch on San Francisco public housing developments have never helped police officers arrest a homicide suspect even though about a quarter of the city’s homicides occur on or near public housing property, city officials say.

Why is that? Could it be because “Nobody monitors the cameras, and the videos are seen only if police specifically request it from San Francisco Housing Authority officials”. No shit.

Another interesting comment:

The cameras have occasionally managed to miss crimes happening in front of them because they were trained in another direction, and footage is particularly grainy at night when most crime occurs, according to police and city officials.

In the digital world we don’t have darkness clouding our detection instruments, but our “cameras” (intrusion detection systems) can be just as grainy and badly positioned. I agree with Schneier when he says that the cameras, who intends to have a scaring affect on criminals, also causes a false sense of security for the inhabitants of San Fransisco. Since we normally don’t disclose that we have “digital cameras”, aka intrusion detection or prevention systems, in the digital world we can pretty much rule out the scare off effect. What about the false sense security?

This years Blackhat featured a talk about reverse engineering IPS signatures. The talk demonstrated that vulnerabilities can be reverse engineered from signatures, which is especially interesting with zero-day signatures. The talk got alot of attention and Tippingpoint, who was the IPS which the condition was demonstrated on, apperently changed their way of distributing their zero-day signatures. I’m looking forward to getting my hands on the talk itself or the slides to get a glimpse of the methodology.