By: Kenneth Barbalace, Published: 2006-03-24, Parent Category: Advertising
In November of 2003 I began researching and developing methods to detect and block users who were blocking banner ads that are displayed on my website EnvironmentalChemistry.com. While I have been aware of and experimented with ad blocking techniques as far back as 1996, I never saw it as a mainstream threat as ad-blocking was not a feature provided by mainstream software vendors. Downloading, installing and configuring these early ad-blocking programs required a level of technical ability well beyond the average user.
In the fall of 2003 I started to realize that ad-blocking capabilities were no longer features provided by obscure or niche software programs, but was beginning to be rolled into mainstream applications like Symantec's Norton Internet Security and ZoneAlarm Pro. Furthermore, in some instances these programs were being installed on pre-built machines and ad-blocking was being turned on by default. As such, many users who were blocking ads were completely unaware that they were blocking ads.
To me, companies like Symantec delivering software that blocked ads by default under the guise of Internet security was not only disingenuous, but it represented a serious long term threat to advertising supported content sites being a viable business model. I felt that if I were to protect my long-term interests, I had to develop countermeasures that defended my interests.
My intention with this article isn't to provide the reader with the exact code necessary to detect and block users who are blocking ads. Rather my intention is to bring the whole ad-blocking issue into perspective and provide theories upon which web publishers can develop their own unique ad-blocking countermeasures. The reason I won't be giving out exact code is that in order for ad-blocking countermeasures to be effective, each site's code must be unique. If everyone were to use the same code to detect and block ad-blocking users, it would create too large of an incentive to expend considerable resources to defeat that one set of countermeasures. By everyone developing unique code sets that function differently, it creates too many targets for those who would like to circumvent ad-blocking countermeasures to reliably overcome.
Contrary to what some would like to believe, ad-blocking software and techniques have been around for almost as long as there have been banner ads. In fact, I remember a program called JunkBuster from back in the 1995-1996 timeframe. Back then ad-blocking was pretty much used exclusively by those with the skill sets required to deploy those early ad-blocking techniques. What really made ad-blocking easier was when the IAB standardized on specific dimensions for banner ads (e.g. 468x60), thus making it much easier to detect ads.
As with so many forms of advertising from early on, banner advertising become an ever escalating screaming match with advertisers assaulting users with ever more obnoxious ads that flashed, jumped, pulsed, etc. in an effort to get the attention of users. The more obnoxious ads became the more desperate users were to find ways to stop the assault on their eyes. Instead of recognizing how annoying their ads were becoming, some advertisers simply tried to find ever more aggressive ways to get around ad-blocking software and force users to pay attention to them.
Eventually this "arms" race lead to the current generation of ad-blocking programs and plug-ins that are highly effective, easy to deploy and easy to maintain. Today, if users really want to, they can surf the web virtually ad-free using any of dozens of ad-blocking solutions. These solutions include extensions for the Firefox web browser like AdBlock, easy to install software based proxies like AdSubract, and security applications like Symantec Norton Internet Security and ZoneAlarm Pro.
To gain insight into just how big of an issue ad-blocking is, I recently crunched all of my server logs for EnvironmentalChemistry.com for the last six months of 2005. I cross referenced the number of daily unique visitors I got against the number of visitors I rerouted to my banner blocking page. I also cross referenced the daily page views reported by my server logs against my Google AdSense stats.
As a point of reference, the stats program I use is AWStats and it has been configured to detect around 400 known robots and automated processes, however I am unable to detect robots that cloak themselves by reporting the UA strings of normal web browsers. Although this browser spoofing will affect the stats I believe the impact is very nominal.
Due to the way I implemented ad-blocking countermeasures on my site, it is only possible to provide a high and low limit as to the percentage of users who are blocking ads not an exact estimate. The low limit was calculated by comparing the total number of visits per month against the number of visits that resulted in the user being redirected to my banner blocking page. Because some ad-blocking methods prevent JavaScripts from running or prevent the JavaScript "onload" event from running correctly, not all users who block ads get redirected to this page. The high limit was calculated by comparing the number of page views that should have displayed a specific Google AdSense ad with the actual number of AdSense ads that were displayed.
| Month | % of Visits Blocked Due to Ad-Blocking | % of Page Views Where AdSense Ads Did Not Load |
| Jun-05 | 3.54% | 12.32% |
| Jul-05 | 3.31% | 11.97% |
| Aug-05 | 3.75% | 9.89% |
| Sep-05 | 3.39% | 9.15% |
| Oct-05 | 2.54% | 10.94% |
| Nov-05 | 3.46% | 9.74% |
| Dec-05 | 4.40% | 10.35% |
When I crunched my logs one figure I wanted to see if I could get a grasp on was what percentage of ad-blocking was due to which browser. The real problem with this is that due to the way logs were processed it is not yet possible to breakdown what percentage of users of a specific browser are blocking ads. It was only possible to show the relative percentage of banner blocking for specific browsers. For instance, in June 2005 Firefox accounted for 25.4% of traffic that was blocking ads. More specifically in June 2005 Firefox accounted for 25.4% of requests for the file "/banblock.html", which is only requested when the user gets redirected because they are blocking ads.
If the percentage of total traffic for a specific browser is lower than the percentage of banner blocking traffic for that browser then users of that browser are blocking ads more frequently than users of other browsers. If the percentage of total traffic for a specific browser is higher than the percentage of banner blocking traffic for that browser then users of that browser tend to block ads less frequently than other users.
For example, Firefox users consistently block ads more frequently than MSIE users. In part this is because of the Adblock extension that users can download for Firefox. One obvious omission from the following list of browser stats is the Opera web browser. Opera was omitted because it consistently generates less than 0.3% of the total traffic for EnvironmentalChemistry.com making any analysis of it statistically irrelevant. One very interesting trend that is disclosed by the table below is that the relative percentage of users blocking ads compared to other users (MSIE users in particular) has been steadily dropping over the past six months even though the total percentage of users using Firefox has climbed modestly over the same period. Why this is happening is anybody's guess.
| Month | Browser | % of Total Traffic | % of Banner Blocking Traffic |
| Jun-05 | Firefox | 7.6% | 25.4% |
| Jun-05 | Mozilla | 1.5% | 1.4% |
| Jun-05 | MSIE | 87.4% | 68.9% |
| Jun-05 | Netscape | 1.3% | 1.3% |
| Jun-05 | Safari | 1.3% | 1.1% |
| Jul-05 | Firefox | 8.3% | 21.8% |
| Jul-05 | Mozilla | 1.6% | 1.1% |
| Jul-05 | MSIE | 86.5% | 73.8% |
| Jul-05 | Netscape | 1.4% | 1.5% |
| Jul-05 | Safari | 1.4% | 1.2% |
| Aug-05 | Firefox | 8.2% | 23.5% |
| Aug-05 | Mozilla | 1.4% | 1.9% |
| Aug-05 | MSIE | 87.3% | 71.9% |
| Aug-05 | Netscape | 1.2% | 1.3% |
| Aug-05 | Safari | 1.3% | 0.8% |
| Sep-05 | Firefox | 8.1% | 14.7% |
| Sep-05 | Mozilla | 0.9% | 0.8% |
| Sep-05 | MSIE | 88.0% | 81.7% |
| Sep-05 | Netscape 0.9% | 1.2% | |
| Sep-05 | Safari | 1.5% | 0.3% |
| Oct-05 | Firefox | 7.6% | 21.7% |
| Oct-05 | Mozilla | 0.7% | 3.1% |
| Oct-05 | MSIE | 88.4% | 72.0% |
| Oct-05 | Netscape | 0.9% | 1.2% |
| Oct-05 | Safari | 1.6% | 0.0% |
| Nov-05 | Firefox | 7.5% | 16.2% |
| Nov-05 | Mozilla | 0.7% | 1.0% |
| Nov-05 | MSIE | 88.2% | 79.0% |
| Nov-05 | Netscape | 0.9% | 1.2% |
| Nov-05 | Safari | 2.0% | 1.3% |
| Dec-05 | Firefox | 8.2% | 16.4% |
| Dec-05 | Mozilla | 0.9% | 0.8% |
| Dec-05 | MSIE | 87.0% | 79.8% |
| Dec-05 | Netscape | 0.8% | 0.6% |
| Dec-05 | Safari | 2.1% | 1.1% |
While there is obviously great room for discussion as to what percentage of users are blocking ads and that percentage changes depending upon the demographics of the site in question, it is fairly obvious that ad-blocking is having a real economic impact on websites.
While there are countless programs promoted that will block ads, they all use a handful of methods to detect and block ads. Most ad-blocking software use a combination of the following methods to detect ads.
Flagging ad delivery servers: Most ads are delivered from specific servers that are dedicated to the delivery of ads (e.g. all AdSense ads come from http://pagead2.googlesyndication.com). Since these servers typically don't deliver normal content, lists of these servers are compiled and distributed in a fashion similar to virus definition updates.
Flagging Keywords in URL: In their infinite wisdom, ad providers make detecting ads very easy by commonly using specific strings within the URL's of ads (e.g. "/banners/"). By simply using regular expressions, ad blocking software can detect and flag ads.
Flagging Specific Object Sizes: While standardizing the sizes of ad-formats made it easier to produce and distribute ads, it also made it easier for ad-blocking software to detect ads by simply flagging objects that are of specific sizes.
Flagging Specific Media Types: Some media types (e.g. flash) are most commonly used for ads and not as commonly for regular content. Thus, these media types can be safely flagged and blocked.
Regardless of software package, once detected, all ads are blocked/hidden using one of the following methods. Without a significant architectural change to the way browsers work there can be no other methods.
Prevent ads from being downloaded from servers: This is the simplest and most commonly used method of blocking ads. Even within web browsers there is an option that prevents the download of images from third-party servers.
Remove HTML source code for the ad from the webpage: This method is commonly used by proxy software. Basically, when the proxy server or firewall (e.g. ZoneAlarm Pro or Proxomitron) detects an ad, the html source for the ad is written out of the webpage before delivering the page to the web browser.
Use CSS to hide ads: With this method, the ad object has its CSS state set to "display:none" or "visibility:hidden". "Display:none" collapses the space taken up by ads, which can mess up the layout of webpages. "Visiblity:none" hides the ad, but leaves "white space" where the ad should be. This preserves the layout of the page, but leaves lots of holes. Sometimes hiding ads via CSS is done in conjunction with preventing the ad from downloading but not always. The theory behind hiding the ad but allowing it to download is that this is harder to detect (which is false) and publishers won't care as the ad still loads and thus is still getting paid.
Replace ad image with a placeholder: An example of this method would be the Flashblock extension for Firefox. While used by Flashblock, this method is not commonly used. For normal image based banner ads, this method may prove to be hard to detect, however, it is also easy to defeat by simply not providing width and height clues in the webpage's HTML and CSS.
The methods I have deployed to detect ad-blocking are JavaScript based and the ad-blocking detection script is called by the "onload" event. While some types of ad-blocking could potentially be detected using server side methods, JavaScript would be required to detect most ad-blocking and ad-blocking countermeasures would not be effective with server side detection.
Were ads pulled from the server correctly? This can be detected in two ways. On the server side, the ads that were supposed to be delivered and those that were actually delivered could be logged on a page request by page request basis. On the client side, JavaScript can be used to detect if the object exists or not. Assuming width and height clues were not provided in the HTML and CSS, the dimensions of the image can also be tested.
Was the HTML code for the ad stripped from the page? This can be tested by wrapping each ad in an additional DIV tag and testing the height of the DIV tag and testing whether the ad object exists or not.
Was the ad hidden from the user? This can be detected by using JavaScript to test what the CSS display and visibility state is for the ad. It could also be tested by testing the dimensions of the ad and surrounding DIV tag.
Once ad-blocking is detected, it may be desirable to block users who are blocking ads. While detecting and redirecting users to a banner blocking page is reasonably straight forward, preventing users from disabling the detection script and circumventing these countermeasures takes a lot more work and requires a holistic approach to the entire site.
An example of a completely integrated detection and blocking script that is currently in use can be found at http://EnvironmentalChemistry.com. Users of this site are not allowed free access to the site's content pages unless its ads are allowed to display properly. If users want ad-free access to the site they are required to purchase an ad-free subscription.
Using a combination of client side and server side tricks, the blocking of users who are blocking ads could be denied access to subsequent pages on a site, rather than simply being redirected as is done on EnvironmentalChemistry.com.
If it is decided to block those who block ads, it is advisable to offer a reasonably priced ad-free subscription option for users. While it is unlikely that this would be a significant source of revenue, it does provide an ad-free option to those users who really want it and it removes the red herring defense "well, I'd pay for ad-free access if they offered it".
The way ads are deployed makes the detecting and blocking of ads way too easy. A lot of things can be done to make ads harder to block.
Change file naming vocabulary: Commonly the URLs of ad graphics contain strings like "/ads/" or "/banners/" which URLs for other graphics files do not have. When possible, avoid using these common strings in the URLs of ads. In fact, ad graphics should be served from the same folders as regular graphics.
Use the same server for ads and other content: Typically ad graphics are served from different servers from the rest of a web site. Thus ad-blocking software only needs to block graphics coming from the ad server to block the ads. By serving the ads from the same server as the regular content, ad-blocking software can not target ads based on where they are served from.
Don't give height and width clues: since some ad-blocking software target ads based on their height and width, do not include width and height clues for ad graphics either in the HTML or CSS. Alternatively change the height or width slightly (e.g. use 469x61 instead of 468x60).
Use randomly fluctuating URL structure: this is a harder trick, but using the rewrite rule in the .htaccess file for Apache, one can add a randomly fluctuating string to an ad graphic's url making it harder to explicitly block an ad file based on its URL. This is particularly useful at thwarting ad-blocking software where users share ad-blocking definitions. Some ad or affiliate networks accomplish this same concept by serving ads from random domains.
Another trick that can be deployed to make ad-blocking software less reliable is designing normal content and images to look more like ads. This will trick ad-blocking software into blocking non-ad content. This will make the use of ad-blocking software a less enjoyable experience because the software will appear to be overly aggressive.
If users feel that these measures are aggressive and become extremely annoyed by obnoxious ads, then they will develop more aggressive ad-blocking technologies. If, however, publishers show a willingness to strike a balance it could reduce the risk of a backlash. By striking a balance, I mean although publishers were requiring users to view ads or subscribe to an ad-free subscription option, publishers would also reframe from deploying the ad formats that most irritate users even if they are the most profitable (e.g. popups, DHTML floating overlays and in some instances flash).
Although I have taken a very aggressive stance against ad-blocking software on my site EnvironmentalChemistry.com, the outcry over my tactics has subsided over the past year. In part I think this was because I did work on addressing the issues that users most objected to with ads and because I make sure users see a direct correlation between the ad revenues I generate and my ability to produce the content that draws them to my site.
This article was written by Kenneth Barbalace. Kenneth Barbalace is the creator and publisher of EnvironmentalChemistry.com, which provides environmental, chemistry & hazardous materials news, information & resources.