Build Your Own UTM With pfSense – Part 1

Photo of author

Tim Higgins



When we last saw Cerberus, the small form factor, low power, high performance IDS firewall, it was chewing through anything the net threw at it. Today’s question is: can Cerberus go for the gold and become a full-fledged Unified Threat Management (UTM) Appliance, capable of providing all of the protection required by a home network, let alone an enterprise network?

Cerberus, as the previous article detailed, is an IDS Firewall built around a mini-ITX 1.8 GHz dual-core Atom and 3 GB of memory, providing three heads of network protection: pfSense, a free open source project, providing standard perimeter firewall protection as part of an overall router, and two pfSense packages: Snort, the premiere open source Intrusion Detection and Prevention rules engine, and IP Blocklist, which uses dynamic categorical lists to block questionable traffic.

To build a capable UTM appliance, we first need to define what Unified Threat Management is. Once we understand that, we’ll step through adding and configuring those services to Cerberus, and finally look whether Cerberus can carry the weight or fall short in either functionality or performance.

What is a UTM Appliance?

The concept of Unified Threat Management is straightforward: on the outer reaches of your network perimeter, you install an appliance that stops all possible threats to your network, an über firewall, as it were. The fact of the matter is that UTM hardware is expected to completely overtake separate network protection hardware.

The problem is there is no single definition of the services required in a UTM appliance. For example, one of the foremost makers of UTM appliances for the enterprise, Endian, lists an entire dense page of functionality. In comparison, Untangle, a small organization UTM, lists only about twenty functions.

So what do they have in common? For our purposes, a UTM appliance is something that offers Intrusion Protection Firewall, Anti-Virus, Anti-Spam, and Traffic Control features. Beyond this core protection, a UTM appliance generally includes some enterprise operation capabilities, such as load balancing, fail-over, and network wide caching and monitoring.

UTM diagram

pfSense can perform all these functions to some extent. To judge how well pfSense meets these UTM requirements, I’ve given a subjective grade to each set of UTM function groups. Once we’ve defined how these functions thwart threats, and how pfSense meets those challenges, we’ll upgrade Cerberus, and see how it performs as a UTM.

Intrusion Detection and Prevention (IDS/IPS)

Snort logoAs detailed in the first article, IDS uses a packet inspection engine in conjunction with a standard NAT firewall to recognize patterns in network traffic, either at the packet level or at the stream level. IDS uses dynamic rules to spot these irregularities, such as protocol vulnerabilities, port scans, Denial of service attacks, and alike.

The vast majority of UTM appliances utilize Snort, the most widely deployed IDS/IPS rules engine. Snort uses rules that are updated regularly from pfSense has wrapped Snort in an easy to install and administer WebGUI package.

Cerberus is already configured for Snort, so we’ll not be covering that as part of the upgrade process. For detailed instructions on how to install and configure Snort, please refer to the previous article.

pfSense Grade: A


The ability to block the Internet’s malicious flora and fauna from infecting network clients is core to any UTM. This is accomplished by inspecting packets for establish virus signatures and virus meta-patterns.

HAVP logopfSense includes the HAVP package: HTTP Anti-Virus Proxy, a transparent proxy that scans all HTTP traffic for malware signatures. HAVP utilizes ClamAV, the open source and community anti-virus engine for Linux and BSD distros.

Naturally, the question of effectiveness is raised when using an open source anti-virus solution versus a commercial product. But is difficult to make a clear determination of effectiveness. Some reports place ClamAV in the top five, others in the bottom five.

There is a dirty little secret in anti-virus detection. Most anti-virus programs are good at detecting known malware. But with the preponderance of free Anti-virus solutions, virus writers are able to craft their code to avoid most prevention solutions, they can test their code before it is released into the wild.

This means that anti-malware solutions effectiveness should really be measured in latency, from the point that they are first seen in play, to when they are added to their respective detection databases. Commercial vendors run network scanners, honeypots, and have dedicated personnel associated with finding the newest threats. ClamAV does not have such resources and hence operates at a disadvantage.

HAVP, as the name implies, is also limited to HTTP traffic. This means that viruses imbedded in files transferred via FTP, HTTPS, and other protocols such as P2P are not examined and would not be detected. Neither are e-mail attachments scanned, which account for one of the largest causes of malware infections.

Because of this, it is important that UTM based anti-virus not be your only malware line of defense. Per client, anti-virus is a critical part of any network’s protection. With so many quality products that can be had at little or no cost, there is no excuse not to run anti-virus on each network host.

Additionally, since it is strongly recommended that you run only one anti-virus application per host, HAVP does have significant utility, because HTTP is one of the largest vectors for infection. HAVP gives you two bites at the apple and offers protection against malware that is targeted at closed systems, such as cell phones and Internet-enabled home theater components.

pfSense Grade: C-

Content Filtering

Content filtering is what it sounds like: the ability to block certain and generally NSFW content from your network. Such content is typically porn, gambling, file sharing, and hacking methods, but can extend to bandwidth-consuming audio/video sites and time-consuming social networking, forum, and blog sites.

Most importantly, it can be used to block IP addresses associated with spamming, malware, and addresses deemed to be compromised in some other way. Unless you have kids, this is the category that is of the most interest to home networks.

pfSense excels at content blocking and offers four different packages for controlling what can come in your front door.

Content Blocking Packages
DNS Blacklist Included functionality uses a static category list Domain blocking by category
Country Block Add-on Package Block entire country access
Squid Guard Add-on Package, works in conjunction with Squid Caching Proxy Server Full Featured URL filter
IP Blocklist Add-on Package, uses frequently updated categorical address lists from Block IP Addresses based on diverse set of lists

Both Country Block and DNS Blacklist are simple. DNS Blacklist, which use a simple list of categories, is a real grab bag and allows the standard blocking of adult and gambling sites, but also astrology, and for some reason, French educational institutes sites (?!?).

IP Blocklist, which had its origins in the P2P peer blocking arena, blocks hosts that perform IP tracking for media companies and associations like the RIAA and the MPAA. It has grown to allow the blocking of spammers, advertising, malware, and other compromised sites. The lists differ significantly in quality; some are excellent, with spot-on targeting, while others seem ill-maintained, and hence have unintentional causalities – for example, one of the adware lists blocks all of CNet.

Squid Guard logoThe real star here is Squid Guard, which works with the caching proxy server Squid. Squid Guard allows for Access Control Lists for specific IPs, with scheduling and user-defined redirect pages. It comes with a built-in blacklist, but also allows the use of community-maintained categorical blacklists. Squid Guard is an ideal solution for café hotspots, schools and libraries.

pfSense Grade: B


Unless you are running a domain out of your home, there is not a lot of call for anti-spam. However, for folks who run a domain’s mailserver, spam is a real problem. The current estimate is that over 75% of all e-mail traversing the net is spam. Spam traffic is a burden on any network, and as previously stated, e-mail accounts for one of the largest vectors for malware infection, either as attachments or through referred malicious web-sites.

pfSense does not currently provide an anti-spam solution. For that solution, you need to drop to the underlying operating system, FREEBSD, which offers numerous packages. There are two significant open source projects for controlling spam: SpamD and SpamAssassin. Notably, in the next release of pfSense, version 2.0, support for SpamAssassin is planned.

FreeBSD LogoThe Perl-based SpamAssassin is a complex spam filtering tool, analyzing the e-mail stream for tell-tale indications that the mail being received isn’t legit. This includes the use of White and Blacklist to vet the e-mail. Beyond filtering, it also can be configured to use ClamAV for malware scanning of the e-mail payload. Depending on your e-mail load, this can be processor intensive.

SpamD takes a much simpler, but clever approach to thwarting Spam. It pretends to be a sendmail-like daemon for mail processing, analyzing the sender against three lists: a white list of approved senders, a black list of known spammers, and a grey list of yet-to-be verified senders.

If on a whitelist, it passes the connection on to the proper mail processing daemon behind the firewall. If it doesn’t know the sender, it responds with a “Please Send Later” message, deferring delivery and adding the sender to the grey list. If the mail is actually resent later, the sender is added to the whitelist, and the mail connection passed on for delivery.

If the sender has been black listed, SpamD tarpits the connection, very slowly and repeatedly asking for details, like a brain-damaged sendmail.

The grey list process counts on the fact that most spam is delivered by hit and run bots, and if delivery fails, the process will just move on. The black list process just screws with the process, slowing down or stopping the ultimate delivery of spam to recipients.

Notably, when it comes to threats, pfSense creates an overlapping field of fire approach with many packages working in conjunction to avert the success of a threat. With spam, Snort provides a set of spam/phishing rules. Country Block content filtering provides a list of the countries most responsible for spam (I personally don’t see a lot of correspondence from Korea, the number one source of spam). IP Blocklist and DNS Blacklist both provide lists for blocking spammers. This is also true of content management where Snort has a set of rules defining inappropriate content. Phrases like “XXX Teen” and other more colorful words can trigger the source address to be blocked.

pfSense Grade: D

Traffic Control

Part of threat management is the ability to control traffic on your network. This includes Quality of Service (QOS) and protocol/application blocking such as P2P, IM, and Gaming or Tor proxy traffic. pfSense doesn’t provide a single point of traffic control. Snort provides protocol blocking – a set of rules that block specific traffic, like P2P.

QOS, the allotting of particular levels of bandwidth to specific applications/hosts or protocols, is accomplished through a Traffic Shaping Wizard that allows you to both prioritize and limit different types or destinations of traffic. The Wizard is very good at simplifying a complex problem, but does not allow a high degree of fine tuning. Additionally, the current version of traffic is limited to single-WAN/LAN prioritization. Version 2.0 of pfSense, now in beta, allows for Multi-WAN/LAN configurations.

The pfSense traffic shaping wizard uses your real world speed to allocate bandwidth, and steps you through a series of pages that allow you to “Shape” specific traffic. These include VOIP, P2P, Gaming, and other application traffic such as HTTP, Instant Messengers, VPN, and Multimedia traffic. You are also allowed to penalize (limit) bandwidth for either a single IP or a Single set of IPs.

Squid logoThe Squid Package is a tunable caching proxy server, which provides both a high speed cache, and the ability to throttle traffic. You can throttle all HTTP traffic, per host traffic, specific traffic by category such as binary or multimedia, or by specific user defined extensions, say avi, mp3, and zip extensions. You can also set maximum upload and download sizes to further limit bandwidth usage..

Another aspect of Traffic Control is the ability to encrypt traffic via a VPN. Three different VPN standards are supported: OpenVPN, IPSec, and PPTP. Under the current version of pfSense, both PPTP and IPSec have NAT limitations, making OpenVPN the most flexible solution. These limitations are well documented and a thumbnail of the issues is covered on the pfSense Capabilities Page.

pfSense Grade: B

Enterprise Capabilities

To paraphrase Doctor Strangelove, “What use is threat management if you don’t have a network?” Safe network access has become indispensable. Any primary network gateway needs to provide for failover, at both the hardware and the provider level.

pfSense provides for hardware failover, network load balancing and failover, and a plethora of ways of monitoring its current and historical status. Hardware failover is handled through synchronized clustering of two separate pfSense boxes, utilizing the pfSense package CARP. Setting up CARP is outside the scope of this article (I don’t have two pfSense boxes, but it appears to be straightforward).

pfSense has built -in Multi-Wan failover and load balancing, utilizing three tiers of cascading gateways: a single load balancer gateway and a gateway for each ISP fail-over point, each having a separate ping heartbeat (say the IPs for Google or Yahoo) that points to the gateway to the ISP. Here is the diagram from the pfSense tutorial.

pfSense multi-WAN

Fail-over is pretty straightforward, active standby is dead simple. The tricky part comes with load balancing, which uses a connection-based simple round-robin algorithm. Quite a few applications/protocols are stateful when it comes to your IP address, such as P2P, games, and IM applications. For each of these you’ll need to set up routing rules that bypass the load balancer and direct the traffic through a particular ISP.

With HTTP connections, pfSense attempts to be sticky, that is, routing the same host through the same ISP, but this is hit and miss. You may see problems with web sites that count on your IP Address not changing, such as cloud based e-mail services and banks.

Regretfully, in the current stable version of pfSense, On-Demand connections, passive standby—like using USB Wi-Fi modems—is not currently supported. But this has been added in version 2.0. Without passive standby, failover is not very attractive to home networks, unless you are willing to incur two ISP bills a month. If you are, then load balancing becomes compelling, even with the routing hassles. Who wants to pay for bandwidth they don’t use?

Enterprise capabilities would not be complete without talking about monitoring, pfSense offers out-of-the-box Syslog and SNMP logging, and several adaptor packages for other protocols, such as RADIUS, NetFlow, and Zabbix protocols. For bandwidth monitoring there is both RRD and a mostly integrated BandwidthHD web display, which breaks out traffic by host IP.

pfSense Grade: C

Closing Thoughts

One important factor that can’t be ignored is that up-to-date content is needed for a UTM appliance to do its job. Without regular updates of IDS rules, host lists, and malware signatures, threat management is no better than a firewall.

For commercial vendors of these appliances, this is a major source of revenue. With pfSense, this content is largely free – making pfSense, with all of its patchwork flaws, very compelling. The value proposition of pfSense is significant. It is free, open, and no expensive subscriptions are needed to protect your network. Free something is better than nothing. So in Part 2, I’ll step you through adding and configuring these UTM features to pfSense.

Related posts

A DIY SSL VPN with SSL-Explorer – Part 1

SSL-based VPNs are fast replacing difficult-to-configure IPsec and PPTP gateways. In Part 1 of a two part series, Phillip Howell shows you how to turn a spare Windows PC into a pretty slick SSL VPN server.

How To Securely Browse from Anywhere using Hamachi and Squid

You can securely browse the web from any public location by using a secure web proxy.

A DIY SSL VPN with SSL-Explorer – Part 2

In Part 2 of his DIY for creating an SSL VPN, Phillip Howell walks through configurations for remote file browsing and desktop access