DCO: Do You Know What Your Network Security Systems are Looking For?

Over the past 3 years, I have been supporting Defensive Cyber Operations (DCO) capabilities for various Department of Defense (DoD) customers, along with an additional 7 years within Network Security Engineering shops. Within the DCO/network security space, there are a myriad of software providers who advertise their Intrusion Detection Systems (IDS) as capable of identifying compromises within your network, and alerting you to the presence of various tactics, techniques and procedures (TTPs).

It sounds great! Just install this software, let it sniff my network, and sleep soundly knowing you are safe from all the cyber threats, right? If only that were the case, I’ve often found there is a gap in understanding what will and what will not be identified by an IDS. There are some key distinctions in the ever-bloated world of buzzwords and promises in cyber.

• Signature/Rules Based
• Heuristic
• Sandbox

These three groups encompass the vast majority of detection systems, and categorizing your detection tools into these categories can help you identify capabilities and gaps within your architecture.

Signature/Rules Based – The most commonly seen IPS/IDS is one that has an internal ruleset, some general best practice match rules, some community-generated rules mapping to known bad IPs, file hashes, urls, etc., and the ability to add rules based on your internal Network Operations Center ()experience. REGEX and YARA are simply formats used to create matching rules against specific attributes. These tools can be very helpful, as long as you understand their function AND limitations. Signature-based systems will not help you find malicious activity that has not been previously observed and quantified into a ruleset.

Heuristic –Heuristic-based IPS/IDS attempt to do what the name implies: learn, or infer from observations, what could potentially be malicious activity. These tools can be a little dicey to define. Unless you are a developer by trade, these systems can appear to be a black box, making decisions based on lengthy scripts that parse network data, inspect headers and payload for whatever the developer-perceived “suspicious” behavior might look like. I know that the tone here seems very cynical, so let me provide a disclaimer; many of these heuristic-based systems use scripts developed by Incident Response Teams (IRT) and professionals, and their logic is nothing to sneeze at. These systems can be challenging to enhance or understand why they make the decisions they do. Today more and more systems within the heuristic category boast of machine learning and artificial intelligence (ML/AI) capabilities to identify malicious activity, and these are powerful tools that will only become stronger as time goes on. Today, however, they have a very high knowledge/training prerequisite that can make them challenging to understand or tune.

Sandbox – This has always been one of the most fascinating system types to me. Relatively early in my career, I learned of a system that, after parsing away all the known malicious/spam traffic via signature-based methods covered above, would create virtual machines on the fly, detonate (open/interact) with objects, and extract the system logs into an “OS change report.” This capability allows a security team to see the exact actions taken by object that passes through the network, with no flags/decision trees prior to being sandboxed. In today’s cyber climate, I believe tools such as these will become increasingly valuable. Object hashes, compromised networks, known bad IP subnets…all those things change, sometimes at a splintering pace. What doesn’t change often, and usually very slowly, is the common end goals of malicious actors (exfiltration, ransomware, denial of service, etc.). Sandbox sensors don’t care how you act  up to and through the front door. They care about what you do in the house.

Sounds like the verdict is in, right? Everyone should trade in their current setup in favor of sandbox sensors! I know some vendors that would be very happy with that assessment…but that is not the key take away here.

The goal of this post is to help organizations and individuals understand the differences between these types of products. Each has its own niche and role that makes it superior to the others in a specific use case. What I hope you take away from this is: if you have deployed a Security Onion IDS on your network, that’s great! But you should be aware of its strengths and weaknesses, abilities and limitations.

What is needed within organizations that require a high level of security and visibility is an orchestra of these capabilities, complimenting each other and covering each other’s gaps. The cyber ihas recognized the value of these different types of systems, and many who offer traditional firewall/IPS systems now offer additional service/license heuristic and sandbox services that pass objects to the vendor’s cloud for additional analysis. These products are often branded as “Next Generation Firewalls.”

Larger organizations that require specific functionality that may not be covered by a single vendor, might build their own security stack that receives forwarded traffic from their environment and then provides a deeper look at traffic that has already been allowed. This method ensures limited impact to performance and availability while still giving security teams an opportunity to catch complex adversaries and quickly identify systems that need to be remediated.

If you are in a position that approves, recommends or creates an IDS/IPS solution, take inventory of your threat surface. If you own or manage a system that has a moderate or high likelihood of advanced persistent threats, you should strongly consider a combination of these tools to take advantage of each system type’s strengths. Understanding your capabilities and limitations is a critical step in maturing your cybersecurity posture. Let’s continue to build, secure and solve together.