Reasons to avoid regular expressions

There are two reasons to avoid regular expression filters. One, they are slower than simple filters. Historically, this hasn’t been the case. The older Adblock used a trivial algorithm for filter lookups, so having a long list of filters would adversely affect performance. In Adblock Plus, a lot of work went into optimizing filter-matching in a large list of simple filters. Jump tables are used to improve filter lookup speeds. For regular expression filters, jump tables aren’t used because requests are matched against every regular expression filter. Wladimir Palant, the author of Adblock Plus, has an excellent blog post that explains the details with graphs.

Two, the expressive nature of regular expressions easily leads to complex, unmaintainable filters. Because the older Adblock encouraged cramming as many rules as possible into a single filter, filters were long and unreadable. Here’s an example of a particularly gnarly filter from Filterset.G:

/[^a-z\d=+%@](?!\d{5,})(\w*\d+x\d)?\d*(show)?(\w{3,}%20|alligator|avs|barter|blog|box|central|context|crystal|d?html|exchange|external|forum|front|fuse|gen|get|house|hover|http|i?frame|inline|instant|live|main|mspace|net|partner|php|popin|primary|provider|realtext|redir\W.*\W|rotated?|secure|side|smart|sponsor|story|text|view|web)?_?ads?(v?((ition|meta|tology3|versal)\.com|(marketplace|rom)\.net|action\.se|bot|brite|broker|bureau|butler|cent(er|ric)|click|client|content|coun(cil|t(er)?)|creative|cycle|data(id)?|engage|entry|er(tis\w+|t(pro)?|ve?r?)|farm|feelgood|force|form|frame(generator)?|gardener|gen|gif|groupid|head|ima?ge?|index|info|js|juggler|layer|legend|link|log|man(ager)?|max|mentor(serve)?|mosaic|net|new||optimi[sz]er|parser|peeps|pic|player|po(ol|pup|sition)|proof|q\.nextag|re(dire?c?t?|mote|volver)|rotator|sale|script|search|sdk|sfac|size|so(lution|nar|urce)|stream|space|srv|stat.*\.asp|sys|(tag)?track|trix|type|view|vt|x\.nu|zone))?s?\d*(status)?\d*(?!\.org)[\W_](?!\w+\.(ac\.|edu)|astra|aware|adurl=|block|login|nl/|sears/|.*(&sbc|\.(wmv|rm)))/

A person can’t tell what that filter is specifically doing with a quick glance. Perhaps more importantly, it makes debugging false positives an absolute nightmare. Of course it’s possible to use clean, sane regular expressions, but their expressive power gives you more rope to hang yourself with. A single regular expression can almost always be replaced with multiple simple filters. In addition to gaining speed, pairing a single rule with a single filter gives the user the benefit of fine-grained control of disabling rules with the green dot.

The real world

There might be some people who say that the need for regular expression filters, no matter how few in number, is inevitable. The success and effectiveness of EasyList, which doesn’t use regular expressions at all, shows this isn’t the case. Fanboy’s AdBlock List shows that lists compatible with Opera and Adblock Plus are possible.

For further information, see how Adblock Plus processes filters and hints for improving speed.

The first thing that jumped out at me was was its page zoom. They fixed the horizontal scrolling bug, making the feature actually usable instead of just being something they can lord over Safari and Firefox for not having. It’s not as glitch-free as Firefox 3’s and Opera’s page zoom, but it’s a step in the right direction. In addition, page zoom now adapts to the Windows DPI setting. Having the font size set to 120 DPI in the Control Panel instead of 96DPI automatically resizes web content to 125% while the page zoom controls remain at 100%. I think it’s great that a web browser respects the user’s choice and largely eliminates the need for users to fiddle with the page zoom settings. The developers still have some work cut out for them because this change absolutely wrecks Sidebar gadgets. Though it isn’t new to IE8, increasing the DPI also makes the toolbars bigger. This is something I wish Firefox would do without having to resort to editing userChrome.css.

Finally, IE8 makes a small change in how URLs are displayed in the location bar. It works the same way as it always has, but now the domain is highlighted. This makes it easier to users who want to find out if the page they’re visiting belongs to the domain they intend to visit instead of some phishing site. Of course SSL certificates remain the best solution for most high-security needs, but this is a welcome change. I think there was some talk of including this functionality in Firefox 3, but I’m not sure if the idea went anywhere. It’s available as an extension, Locationbar², but it suffers from a problem common to most Firefox extensions: insensible defaults and too many configuration options.

Above is a screenshot of how I like my Firefox to be configured. Notice the larger tab size that can be achieved by editing userChrome.css.

In addition to the t-shirt, the people at Opera sent me a lanyard, a couple of pens, and a handwritten note penned by Espen André Øverdahl. To be honest, the cheap quality of the swag disappointed me. But the handwritten note thanking me was a nice touch.

For a time, Opera fulfilled my daily browsing needs instead of the much-beloved Firefox. Drawn into its snappiness and very slick page zoom, I could not bear fiddling with the dysfunctional text zoom in Firefox on my high-resolution monitor. But soon aftewards, Mozilla released Firefox 3 Beta 1 to recapture my heart. It features the long-anticipated page zoom, and does one on Opera better by remembering zoom levels for each hostname. So Firefox remains my primary browser. But Opera still commands my respect.

There are a few DirectConnect clients on Linux, but they’re not worth bothering to use. Most of them aren’t maintained anymore and almost all aren’t compatitble with the latest DC++ which most people use. The sheer popularity of DC++ means that one must use a program based on its core. There are two ways to do this. One can run DC++ through Wine or compile LinuxDC++ (linuxdcpp) from source.

The easiest and future-proof method is running DC++ through Wine. There are many tutorials on the web that can help one set up Wine and run Windows programs. The benefit to this approach is that one isn’t at the mercy of another developer to update the program to the latest version. One can stay up-to-date by downloading the latest DC++ release. The downside to using Wine is that the program looks unattractive and incoherent with the operating system. As noted in the Wine Application DB, there are a few application settings to tweak to get it to run properly. Also, as of the time of this post, the version of Wine in the Ubuntu repositories has a bug which won’t kill the process when closing DC++.

The alternate way to use DirectConnect is to build LinuxDC++ (linuxdcpp) from CVS, a scary proposition for normal users. Fortunately, instructions for Ubuntu are posted in their forums making accessible to the people. Using LinuxDC++ means that the application feels in place with the operating system. Of course, one is now at the mercy of the developer of LinuxDC++ for updates to synchronize the core with DC++.

Choosing which route to take is an exercise up to the reader. Reflecting back on my experience, I would run DC++ with Wine if I hadn’t already compiled LinuxDC++. But for now, either method is acceptable.

In theory, WordPress receives If-modified-since and If-none-match headers, processes them, and responds with an appropriate 200 OK or 304 Not Modified header. Bandwidth would be saved if it matches 304 Not Modified because the content wouldn’t be sent. However, in practice, WordPress does not correctly process the If-none-match header due to some PHP and WordPress weirdness. That means content is sent with a 200 OK header no matter what.

One solution would be to not send the ETag header at all. This would work with some feed readers such as Firefox’s Live Bookmarks feature. The drawback is that it wouldn’t work with feed syndicators such as Bloglines and Facebook because they check both the Last-modified and the ETag headers.

The other, preferred solution is to fix the processing of If-none-match headers with a quick and easy hack. In WordPress 2.0.5, open up wp-includes/classes.php and go to line 1640. Replace if (isset($_SERVER['HTTP_IF_NONE_MATCH'])) $client_etag = stripslashes($_SERVER['HTTP_IF_NONE_MATCH']); with if (isset($_SERVER['HTTP_IF_NONE_MATCH'])) $client_etag = stripslashes(stripslashes($_SERVER['HTTP_IF_NONE_MATCH'])); Now your bandwidth usage for WordPress feeds should be reduced. Don’t forget that you’ll have to do this every time you upgrade WordPress.

Update: It seems that they have applied this patch for the ETag bug for release in WordPress 2.1. Users of 2.0.x will still have to manually apply it.

Disclosure: I actively contribute to EasyList development.

A little background

In the original Adblock, the user had to create filters by themselves to block advertisements. Filtersets like Filterset.G came on the scene so that people could import preconfigured filters that block ads on a majority of websites. Because of the fluid nature of the web, filtersets needed to be updated from time to time. Filterset.G Updater, a seperate extension created by Reid Rankin and Michael McDonald, was the solution to that problem. Adblock Plus further refined the concept with filter subscriptions. It automatically updates filtersets instead of a relying on a seperate extension. Today, Filterset.G and EasyList are some of the most popular filter lists.

Testing configuration

All effectiveness tests were performed with Firefox 2.0 and Adblock Plus 0.7.2.2. Filterset.G 2006-10-26a was paired with Filterset.G-Whitelist-Beta wb-2006-10-26a. EasyList 04Nov2006 was tested both on its own and also paired with EasyElement 31Oct2006. Why didn’t I use Adblock 0.5.3.043? Because as far as anyone can tell, the project is dead. The project has not had a CVS commit for at least 180 days, the website has not been updated with the latest version, the developers don’t post in the forums, the latest xpi has the maxVersion set to Firefox 2.0a, and AMO had to specially grant Michael McDonald upload privileges so that a security vulnerability could be fixed.

Effectiveness tests

Screenshots are provided for comparison. The first thumbnail is Filterset.G. The second one is EasyList. The third one is EasyList paired with EasyElement. Hover over the thumbnails to see which filter list it is. Sorry about the crappy jpegs. The pngs were too big.

MapQuest.com

Filterset.G and EasyList are equally effective at removing the ads. EasyElement goes one step further by removing a few of the text ads, although it is bit too aggressive in this case.

NYTimes.com

Filterset.G and EasyList are equally effective at removing the ads. Filterset.G has a minor issue with a few images related to layout, but it’s no big deal. EasyElement goes one step further by removing text ads.

PCWorld.com

Filterset.G and EasyList are equally effective at removing the ads. EasyElement works well to cover up the ugly spaces that blocked ads leave behind. It’s so good that you can’t even tell where on the page the ads would have been.

People.com

Filterset.G and EasyList are equally effective at removing the ads. EasyElement goes one step further by removing the cruft that the blocked ads leave behind.

Slashdot.org

Filterset.G and EasyList are equally effective at removing the ads. EasyElement goes one step further by removing a little bit of white space, but it’s not a big improvement in this case.

TomsHardware.com

Filterset.G and EasyList are equally effective at removing the ads. EasyElement shows no difference.

Weather.com

Filterset.G and EasyList are equally effective at removing the ads. EasyElement shows no difference.

Finance.Yahoo.com

Filterset.G and EasyList are equally effective at removing the ads. EasyElement goes one step further by removing the space taken up by the ads.

Performance

I contacted the author of Adblock Plus, Wladimir Palant, to get some numbers. The processing time for Filterset.G clocked in at 65ms. The processing time for EasyList clocked in at 27ms. As you can see, Adblock Plus processes EasyList more than twice as as fast as Filterset.G. Using EasyElement in addition to EasyList adds a negligible performance hit. Performance for Filterset.G can be markedly improved by merely converting its regular expression filters to simple filters, but G has chosen not to do so. More information about how Adblock Plus’s algorithms work is available here.

Conclusion

Top honors go to EasyList and EasyElement. Both Filterset.G and EasyList are similarly effective at blocking advertisements. However, EasyList paired with EasyElement gets rid of much of the cruft left over from blocked ads while Filterset.G does not. Also, EasyList performs twice as fast as Filterset.G. Some people may balk at the idea of a mere 38ms improvement, but the difference shows how much effort the filter list maintainers put into making their filter lists better. You can always expect a quick reply from Rick, the maintainer of EasyList, if you post in his forum. Contrast this position with G who refuses to publish a static URL for his list or convert his filters from regular expressions. You can see why I recommend using EasyList & EasyElement.

Update: Wladamir Palant, the developer of Adblock Plus, recommends users to switch away from Filterset.G.

Besides its mass appeal to advertisement-weary web surfers, Adblock (Plus) is different from the rest because it allows users to block a majority of ads by simply importing filtersets (or subscribing to them in the case of Adblock Plus). A few experts create lists of filters that can be consumed by a significant share of internet users–not just the geeks. The average user doesn’t need to fiddle around with creating filters because the work has already been done for them. This is what allows an extension to become more popular than with its enthusiasts and cross into the mainstream. Look at the AMO top ten where Adblock (Plus) and NoScript, an extension which blocks JavaScript on websites unless whitelisted by the user through a single action, are the the only content-modifying extensions listed there.

The lack of subscription options is what keeps me away from adopting other extensions that modify web content. If they want break out of their current statuses as mere curiosities for the developer and geek crowd, one of the steps they should take is to follow the Adblock (Plus) approach of easy mass consumption by way of subscription. Make it so I can set up an extension once and have it working without my constantly having to set up scripts. Script repositories such as Userscripts.org and Userstyles.org is a great step in the right direction, but they are still time-consuming and effort-intensive. Can you imagine having Adblock (Plus) greenhorns craft individual lists of filters to block all the ads they come across? They wouldn’t be of the same caliber as the well-established filtersets, and most users wouldn’t even bother creating such lists.

Adblock Plus is a special extension; it has a lot of great things going for it such as a great developer, an open community, and features that focus on an easy user experience. What I think should be emphasized when promoting Adblock Plus is filter subscriptions. Having Adblock Plus pop up a list of filter subscriptions when first run was a great addition. Maybe Adblock Plus should depart from the old Adblock interface paradigm and separate the filters from the preferences panel. The new preferences panel would devote space to filter subscriptions and options. The filters area of the current preferences panel along with the list of blockable items could be implemented in a different window, one that is accessible by a button in the preferences panel or from the Adblock Plus menu. Though what I put forth is radical, it’s only an idea to generate some discussion, not an expectation. I’m sure Wladimir Palant is busy with his life and I truly appreciate his work. Adblock Plus in its current form is a fantastically implemented extension.

EasyList works best with Adblock Plus. They’re getting some impressive numbers these days. Now that the developer of Filterset.G has appeared to stop updating his filterset and Adblock (always) seems stagnant, Adblock Plus and EasyList are poised to become a dominant force in the ad blocking community. Even if that never quite happens, I think Adblock Plus and EasyList will be highly-praised and recommended because they are very polished and easier to use than their more popular counterparts.

I am learning some skills such as writing regular expessions and inspecting pages with FireBug, which increases my geek cred. The skills and tricks I learn are being put into the development of my own personal Filtersetb Corset. Check it out. You’ll see my recommendations for the best way to block advertisements if you use Mozilla Firefox.