What Is an Intrusion Detection System?

Headline-grabbing attacks, like the Colonial Pipeline hack that shut down a major US pipeline for over a week, demonstrate their potential for major destruction. In 2020, in the US alone, organizations paid an estimated $350 million for ransomware attacks. For each major breach, hundreds of attacks devastate smaller businesses and their customers.

Firewalls and anti-malware programs alone are not enough to protect an entire network from an attack. A well-rounded security strategy should also include an intrusion detection system (IDS) that pinpoints suspicious traffic once it passes the firewall and enters the network.

This article is an introduction to intrusion detection systems and the role IDSes play in network security. Read on to learn how these systems work and why they are vital in preventing costly downtime and data breaches.

What Is an Intrusion Detection System (IDS)?

An intrusion detection system (IDS) is an app or device that monitors inbound and outbound network traffic, continuously analyzing activity for changes in patterns, and alerts an administrator when it detects unusual behavior. An administrator then reviews alarms and takes actions to remove the threat.

For example, an IDS might inspect the data carried by network traffic to see if it contains known malware or other malicious content. If it detects this type of threat, it sends an alert to your security team so they can investigate and remediate it. Once your team receives the alert, they must act quickly to prevent an attack from taking over the system. 

To ensure that an IDS doesn’t slow down network performance, these solutions often use a switched port analyzer (SPAN) or test-access port (TAP) to analyze a copy of the inline data traffic. However, they don’t block threats once they enter the network, as intrusion prevention systems do.

Regardless of whether you set up a physical device or an IDS program, the system can:

• Recognize attack patterns within network packets.
• Monitor user behavior.
• Identify abnormal traffic activity.
• Ensure user and system activity do not go against security policies.

The info from an intrusion detection system can also help the security team to:

• Audit the network for vulnerabilities and poor configurations.
• Assess the integrity of critical systems and files.
• Create more effective controls and incident responses.
• Analyze the quantity and types of cyber threats attacking the network.

Cybersecurity benefits aside, an IDS also helps achieve regularity compliance. Greater network visibility and better logging ensure network operations stay in line with all relevant regulations.

Goals of Intrusion Detection Systems

A firewall alone doesn’t provide adequate protection against modern cyber threats. Malware and other malicious content are often delivered using legitimate types of traffic, such as email, or web traffic. An IDS provides the ability to inspect the contents of these communications and identify any malware that they might contain.

The main goal of an IDS is to detect anomalies before hackers complete their objective. Once the system detects a threat, the IDS informs the IT staff and provides the following info about the danger:

• The source address of the intrusion.
• Target and victim addresses.
• The type of threat.

The secondary goal of an intrusion detection system is to observe intruders and identify:

• What resources attackers try to access.
• How hackers try to bypass security controls.
• What types of cyberattacks intruders initiate.

The company’s security operations center (SOC) and analysts can use this info to improve the network security strategy.

Anomaly detection and reporting are the two primary functions of an intrusion detection system. However, some detection systems can respond to malicious activity, such as automatically blocking an IP address or shutting down access to sensitive files. Systems with these response capabilities are intrusion prevention systems (IPSs).

How Do Intrusion Detection Systems Work?

An IDS monitors traffic to and from all devices on a network. The system operates behind a firewall as a secondary filter for malicious packets and primarily looks for two suspicious clues:

• Signatures of known attacks.
• Deviations from regular activity.

An intrusion detection system typically relies on pattern correlation to identify threats. This method allows an IDS to compare network packets to a database with signatures of known cyberattacks. The most common attacks an IDS can flag with pattern correlation are:

• Malware (worms, ransomware, trojans, viruses, bots, etc.).
• Scanning attacks that send packets to the network to gather info about open or closed ports, types of permitted traffic, active hosts, and software versions.
• Asymmetric routing that sends a malicious packet and bypasses security controls with different entry and exit routes.
• Buffer overflow attacks that replace database content with malicious executable files.
• Protocol-specific attacks that target a specific protocol (ICMP, TCP, ARP, etc.).
• Traffic flooding breaches that overload the network, such as a DDoS attack.

Once an IDS discovers an anomaly, the system flags the issue and raises the alarm. The alert can range from a simple note in an audit log to an urgent message to an IT admin. The team then troubleshoots the problem and identifies the root cause of the issue.

What Are the Types of Intrusion Detection Systems?

There are two main types of IDSes based on where the security team sets them up:

• Network intrusion detection system (NIDS).
• Host intrusion detection system (HIDS).

The way an intrusion detection system detects suspicious activity also allows us to define two categories:

• A signature-based intrusion detection system (SIDS).
• An anomaly-based intrusion detection system (AIDS).

Depending on your use case and budget, you can deploy a NIDS or HIDS or rely on both main IDS types. The same applies to detection models as many teams set up a hybrid system with SIDS and AIDS capabilities.

Before you determine a strategy, you need to understand the differences between IDS types and how they complement each other. Let us look at each of the four main IDS types, their pros and cons, and when to use them.

Network Intrusion Detection System (NIDS)

A network-based intrusion detection system monitors and analyzes traffic coming to and from all network devices. A NIDS operates from a strategic point (or points, if you deploy multiple detection systems) within the network, typically at data chokepoints.

Pros of a NIDS:

• Provides IDS security across the entire network.
• A few strategically placed NIDSes can monitor an enterprise-size network.
• A passive device that does not compromise network availability or throughput.
• Relatively easy to secure and hide from intruders.
• Covers networks parts where traffic is most vulnerable.

Cons of a NIDS:

• Expensive to set up.
• If a NIDS must monitor an extensive or busy network, the system can suffer from low specificity and an occasional unnoticed breach.
• Detecting threats within encrypted traffic can be problematic.
• Typically not an ideal fit with switch-based networks.

Host Intrusion Detection System (HIDS)

A HIDS operates from a specific endpoint where it monitors network traffic and system logs to and from a single device.

This type of IDS security relies on regular snapshots, file sets that capture the entire system’s state. When the system takes a snapshot, the IDS compares it with the previous state and checks for missing or altered files or settings.

Pros of a HIDS

• Offers deep visibility into the host device and its activity (changes to the configuration, permissions, files, registry, etc.).
• An excellent second line of defense against a malicious packet a NIDS failed to detect.
• Good at detecting packets originating from inside the organization, such as unauthorized changes to files from a system console.
• Effective at detecting and preventing software integrity breaches.
• Better at analyzing encrypted traffic than a NIDS due to less packets.
• Far cheaper than setting up a NIDS.

Cons of a HIDS

• Limited visibility as the system only monitors one device.
• Less available context for decision-making.
• Hard to manage for large companies as the team needs to configure and handle info for every host.
• More visible to attackers than a NIDS.
• Not good at detecting network scans or other network-wide surveillance attacks.

Signature-Based Intrusion Detection System (SIDS)

A SIDS monitors packets moving through a network and compares them to a database of known attack signatures or attributes. This common type of IDS security looks for specific patterns, such as byte or instruction sequences.

Pros of a SIDS

• Works well against attackers using known attack signatures.
• Helpful for discovering low-skill attack attempts.
• Effective at monitoring inbound network traffic.
• Can efficiently process a high volume of network traffic.

Cons of a SIDS

• Cannot identify a breach without a specific signature in the threat database.
• A savvy hacker can modify an attack to avoid matching known signatures, such as changing lowercase to uppercase letters or converting a symbol to its character code.
• Requires regular updates of the threat database to keep the system up to date with the latest risks.

Anomaly-Based Intrusion Detection System (AIDS)

An AIDS monitors ongoing network traffic and analyzes patterns against a baseline. It goes beyond the attack signature model and detects malicious behavior patterns instead of specific data patterns.

This type of IDS uses machine learning to establish a baseline of expected system behavior (trust model) in terms of bandwidth, protocols, ports, and device usage. The system can then compare any new behavior to verified trust models and discover unknown attacks a signature-based IDS cannot identify.

For example, someone in the Sales department trying to access the website’s backend for the first time may not be a red flag for a SIDS. For an anomaly-based setup, however, a person trying to access a sensitive system for the first time is a cause for investigation.

Pros of an AIDS

• Can detect signs of unknown attack types and novel threats.
• Relies on machine learning and AI to establish a model of trustworthy behavior.

Cons of an AIDS

• Complex to manage.
• Requires more processing resources than a signature-based IDS.
• High amounts of alarms can overwhelm admins.

IDS Strengths and Limitations

The obvious strength of deploying an IDS is the critical insight into network activity. The early detection of unusual behavior helps minimize the risk of cyber attacks and ensure better network health overall.

Using an IDS to protect a network is a valid strategy to boost security. When paired with a robust anti-malware program and firewall, an IDS ensures the team:

• Stays ahead of a large percentage of cyber problems, whether caused by a malicious actor, accident, or error.
• Does not need to comb through thousands of system logs for critical info.
• Can reliably enforce company’s security policies at the network level.

IDSes (and even IPSes) are also becoming cheaper and easier to administer, so even SMBs with smaller budgets and less IT staff can rely on this strategy. Despite all the benefits, however, IDSes also have some unique challenges:

• Avoiding an IDS is the priority for a successful attack, making these systems the go-to target for hackers.
• Detecting malicious activity within encrypted traffic is a common issue.
• An IDS can be less effective in a network with high amounts of traffic.
• Even the best system can have problems recognizing signs of a novel attack.
• IDS monitors north-south traffic, it does not provide insight into east-west traffic.

The biggest challenge of an IDS is avoiding mistakes as even the best system can:

• Raise the alarm for something that is not an attack (a false positive).
• Fail to raise the alarm when there is a real threat (a false negative).

Too many false positives mean the IT team will be less confident of the IDS’s warnings. False negatives, however, mean that malicious packets are entering the network without raising an alarm, so an oversensitive IDS is always a better option.

IDS Best Practices

Once you know what IDS type and detection model you need to set up, ensure your strategy follows these best practices:

Train IT Staff. Ensure the team setting up the IDS has a thorough understanding of your device inventory and each machine’s role.

Determine a Baseline. To ensure your IDS detects normal from abnormal behaviors, establish a baseline so you know what’s on your network. Keep in mind that each network differs in the type of traffic it carries. Defining a clear initial baseline helps prevent false positives and false negatives.

IDS Deployment. Deploy the IDS at the highest point of visibility to not overwhelm the system with data. Ideally, place the IDS at the edge of your network, behind the firewall. Install multiple IDSes across the network if you need to deal with intra-host traffic. The right choice of system and deployment location depends on the network and security goals.

Tune the IDS to the Network. Change the default settings of the IDS only where it makes sense for the network. Configure the IDS to accommodate all devices, applications, ports, protocols, and security points on the network. By customizing the configuration to apply to your network infrastructure, you create a solid base for detection. 

Set up Stealth Mode. Set the IDS up to run in stealth mode to make the system hard to detect for malicious actors. The simplest way to do so is to ensure the IDS has two network interfaces, one for the network and the other for generating alerts. The IDS should use the monitored interface as input only.

Test the IDS. Test the IDS to ensure it detects potential threats and responds to them properly. Use test datasets or, even better, have security professionals do a penetration test (pen test). Run these tests regularly to make sure everything continues to work as expected. Over time, evolve your testing approach to keep up with changes in the types of attacks that can occur. 

Balance False Positives and Negatives. Be careful not to over-tune your IDS or otherwise misconfigure it, so you don’t create false positives or false negatives. Too many of either can overwhelm your IT and security teams and even put your organization at greater risk of an attack. Combine NIDS setups and network segmentation to make detections more effective and easier to manage.

Investigate and Respond to Incidents. Define an incident response plan ready to act on. This plan must include skilled security personnel who know how to respond quickly and effectively with minimal disruption to daily operations and impact to your organization. If your organization must comply with certain industry requirements, such as HIPAA, GDPR, or SOC 2, define proper controls in place and follow established protocols. Consider adding a secondary analysis platform to analyze threats after the IDS raises the alarm.

Update the Threat Database Regularly. Once the IDS is up and running, your team should continually update the threat database to keep the system effective. Ensure all your IDSes and threat databases follow the principle of zero-trust security.

Do Not Overlook the Value of IDS Security

A high-quality IDS (or IPS) is vital to maintaining acceptable levels of network security. An IDS only detects threats and might not catch every potential one. Therefore, it’s not enough on its own to prevent attacks and protect your organization from them.

Rather, an IDS is part of your overarching security strategy. Besides having the right security tools in place, you need to ensure your employees—your first line of defense—know how to keep your organization, information, and assets safe. That defense begins with an effective cybersecurity awareness program. In return, they’ll have greater confidence in knowing how to react and respond to them, as well as minimize risks to your business and your customers.

This article was written with the help of Eyal Katz, head of mvpGrow and a cybersecurity marketing expert.