What is Leeching?

Leeching. Inline linking. Hotlinking. Bandwidth theft. Sometimes it is even called direct linking or confused with deep linking. Whatever the term, if a third party site is requesting your files and presenting them on their Web site, you are paying for the bandwidth while they use your content as their own.

How Does this Bandwidth Theft Occur?

In a classic example of content leeching, an online ad (i.e. on eBay) is created with an unauthorized image served from a manufacturer’s site. This arrangement is good for the eBay user, but bad for the original manufacturer’s bandwidth bill – they serve the image but get no benefit from the transfer.

Another common image leeching faux pas is internet bloggers using directly linked images from another site without uploading them to their own server or host.

Not only is this a strain on your bandwidth, but in many cases this is an infringement on your copyright as well. It’s a safe bet to assume that if they are using your bandwidth they are most likely using your images without your permission as well.

How do you Stop Leeching?

If you are working in an Apache server environment one way that you can block unauthorized access to your files would be to add the following to your .htaccess file:

RewriteEngine On
RewriteCond %{HTTP_REFERER} !^http://(.+\.)?example\.com/ [NC]
RewriteCond %{HTTP_REFERER} !^$
RewriteRule .*\.(jpe?g|gif|bmp|png)$ /images/nohotlink.jpeg [L]

The basic idea here is that we block requests, sending them to the “nohotlink” image if they don’t include the Referer header that points to our domain, in this case “example.com.”   This is a simple check but interestingly it knocks down most bandwidth thieves.

It is possible to check other headers as well, for example we might look to see if there is a User-Agent header and deny it in a similar fashion as well.  We might expand the scheme to try to address the particular values of a user-agent but this can be quite troublesome to keep up to date.

Microsoft IIS doesn’t have URL rewriting technology built-in, but there are solutions out there including Microsoft’s URL Rewrite module for IIS7+ . However, rewriting solutions really don’t properly address the leeching problem, and thus the rules found in our LinkDeny product should be employed.

Taking it a Step Further

LinkDeny can block using simple checks like header values; particularly the obvious Referer and User-Agent headers. LinkDeny can also enforce that HTTP requests are proper in their minimal number of headers, such as required Host header for HTTP/1.1, their style and so forth.  Basically the idea is that if people want to use bots to steal content they have to get the details down otherwise we will swat the request.

We can certainly go a bit further in limiting the types of people.  For example, commonly we can spot people who are abusing the site and stealing content and just add an IP block.  Or we may think the reverse and disallow everyone but say a few trusted IP partners. Using GeoIP resolution we can even expand this thinking to geographic regions when we see continued abuse coming from particular locales.  We can add such checks into a module or into application logic.

However, the wily content thief will do their best to imitate user-agents correctly and bounce off proxy servers to avoid detection.  We have to assume that these thieves will not be swatted down by simple checks so we limit our exposure by limiting access.  Consider if we have pieces of content like /books/manual.pdf or /images/logo.gif it is quite easy for the unscrupulous to automate a bot or even script a real browser to fetch these objects.  However, we can up the ante by changing the location or the required credentials to access the object regularly.

First let’s consider the use of a cookie. We could issue cookies in our site and require that a user send a cookie to fetch dependent images.  This alone will knock down many bots who don’t accept cookies.  However, our aim here is defeating those people who know all these things and use browser automation.  In this case they will likely accept cookies and will record a script to fetch the desired object.  Now we can foil these folks by making the cookie be short lived and reissuing allowed cookies very often.  So now their script may work for a few minutes but it fails later.  If they re-record it will fail again as soon as the cookie rotates.

Obviously it would seem that this is defeatable simply by visiting the site properly when you want to get the object and that’s ok.  Remember the point of anti-leeching isn’t to lock out legitimate browser access it is to lock out those who don’t play by the rules.

We can take time limitation and make it really hard to deal with by not only requiring a changing cookie but changing the very URLs of the dependent objects over time.  Thus logo.gif might be logo123fdf2345.gif one moment and logo1235234ss.gif another.  Obviously our pages have to reference some sort of token replacement system to employ such a measure but it makes scripting access to the object quite annoying.

Conclusion

All of these anti-leeching schemes can be defeated, just as all locks online and off can be broken.  Ultimately you get into the game of dealing with anti-automation and injecting CAPTCHAs for people to fill out.  These too can be defeated of course, but that doesn’t mean we should just give up. If that is your attitude we suggest you avoid locking your house or car because frankly those employ easily breakable security mechanisms as well!

Whether you roll your own anti-leeching feature or use a product like LinkDeny, protecting your site content from leeches will save you bandwidth, and money. You won’t be able to stop everyone from trying to steal your content, but you can however make a sizeable dent in just how much in how much they do steal!

~ Port80

<img src=”swineflu.jpg” oncontextmenu=”alert(’Copyright 2009 Big News Agency’); return false;”>

However, this is easily defeated. 

You might even try a trick where you put a transparent pixel image over a protected image.  Then the user can access their context menu but when they save an image they get some blank item. An example of this technique can be found on David Walsh’s Programming Blog.  While this scheme is actually employed at a few photo sharing sites it is silly and easily broken, but still somewhat useful for those with no other means of content protection.

Now let’s be real.  If the person wants your image it is easy to go spelunking in your cache and grab it. You could use some interesting tricks with Flash and Java applets to keep the actual bits of the image out of cache, but then you are starting to get into to the “what’s the point” zone.  (Image Stealer): “I’ll show them, I’ll just screen capture that baby.”

So we’ve seen this pattern for image protection for years, but what we haven’t seen is this particular level of silliness in content protection.  Sitepoint exposes some folks who actually make their content difficult to highlight because they use script to put it together dynamically on screen.  Of course this causes SEO trouble, accessibility issues, and just pisses users off.  But hey why stop there… we can use Flash or something to load text in and make it so we have to print and that’s it. 

Snark aside, we really do believe site owners should have control over their sites, but let’s understand that when you start angering your legitimate users you’ve likely gone too far.

Port80

Recently there has been growing interest in various schemes for request bundling. For example, a way to use data URIs in most browsers was shown here: http://www.hedgerwow.com/360/dhtml/base64-image/code.txt. It is ugly, but mostly works. In the past we have seen other folks use a variety of client and server-side schemes to bring inlined images and other assets to the masses - mostly with little success. That’s too bad.

These are great efforts and request bundling is the proper solution for ultimately addressing speed problems as you go beyond Gzip, Caching, and other commonly known methods. For most higher bandwidth users it is the number of requests not byte count that is killing their performance.

We like ingenuity here but honestly Digg’s effort is well… a hack and a half. We can safely say that it is going to blow up for a number of reasons we are directly aware of, so we’ll enumerate.

Data URIs Close But Yet So Far

The Digg scheme relies on parsing the incoming multipart stream and creating data URIs (http://en.wikipedia.org/wiki/Data_URI_scheme) out of it. Older Internet Explorer browsers do not support data URIs. You can try to hack you way around this, as recently shown, using object and MHTML to stuff things in, but testing is showing that approach to be problematic. Even with IE8, which supports dataURIs, you have significant length limitations - 32K for in-page objects and far less for raw URLs.

Multipart Support in Ajax - Rare and Inconsistent

Multiparts with XHRs are not at all consistently supported across browsers. Test it yourself and you’ll see what we mean. Even amongst the supporting ones you are going to get quirks. This is discussed in our CEO’s Ajax book, chapter 3. Worse yet if you are trying to deal with partial streams as you go along; that is also a black art and problematic, particularly in Internet Explorer. The comments in Digg’s code reveal they are discovering that the XHR object varies from version to version in IE. It isn’t commonly known, but the XMLHttpRequest object is actually pretty inconsistent in its details from browser to browser and version to version.

Cachability

A huge problem with many Ajax solutions is caching. It is possible in principle to work with the browser cache or better yet even add an Ajax cache. But most Ajax performance schemes like Digg’s fail in practice because the end user cache is not populated properly–unless you plan on using the dataURIs as the cached item everywhere else, which is pretty unlikely. In order to still use the cache, these implementations must often fire off subsequent requests even after the first page load to populate it.

Right Idea - Wrong Solution

To be clear, we think that an initial, bundled request of dependent assets is a GREAT idea. However, it has been a great idea for years and many of us have tried to do it, but the challenge is all in the execution. This way of doing it is not the right way… at least for the long haul.

Ultimately, the major browser vendors (i.e. Internet Explorer, Firefox, Chrome) need to facilitate this type of performance optimization, by creating a JAR file, bundle, archive, or something, so we can negotiate a large initial request and have our proper caching too. We just can’t be dependent on JavaScript for page composing duties if we want top-notch performance. Even with the massive improvements in JavaScript speed via browser engines like Google’s V8, WebKit’s SquirrelFish, and Firefox’s Tracemonkey, any JavaScript page composing is going to be smoked by a native solution. JavaScript is a great development environment for Web 2.0, RIA type applications. But fixing browser plumbing with JavaScript is just not the way to go for permanent performance gains.

We’ll be happy to be wrong, but having seen this before we kind of doubt it. This is in fact worn ground: Delta encoding with JavaScript was tried well over 8 years ago and it just isn’t something that reliably works in scale.

The good news is that the native browser changes in this area are coming–just wait! We are about to show one that is relatively newly shipping.

- P80

Note: We should note that Digg’s interest in trying to control page paint order is quite appropriate, since the perception of speed comes not just from downloading but how the page paints.

{follow us on Twitter @port80software}

“At issue are sites that harbor so-called cross-site scripting (XSS) vulnerabilities, which occur when Web sites accept input from a user usually from something like a search box or e-mail form but do not prevent users from entering malicious code or other instructions.”

XSSed.com, a site which catalogs XSS flaws on the web, has found more than 13,000 Web pages that hosted XSS vulnerabilities. The XSSed project was created in early February 2007 by Kevin Fernandez and Dimitris Pagkalos. It provides information on all things related to cross-site scripting vulnerabilities and is the largest online archive of XSS vulnerable websites.

“XSS flaws are some of the most common Web site vulnerabilities, but they are also usually fairly simple to fix. If your site is listed on xssed.com, or you’d simply like to know more about how to make sure your site isn’t contributing to the problem, check out this primer from the Open Web Applications Security Project (OWASP). While you’re there, you might want to take a look at some of the other best-practices documents they have available.”

XSS relies on trust. Blindly trusting user input and posting it back to a site, particularly if it includes HTML or JavaScript, is a certain recipe for trouble. If you have access to your Web application code make sure you sanitize all inputs for malicious script based payload. If you don’t have access to code or need an extra layer to make sure nothing can get through, then a Web Application Firewall such as one of the ServerDefender family is in order.

Shannon @ Port80

{follow us on Twitter @port80software.com}

Since so many of the Port80 software tools are focused on security issues, we’re constantly keeping an eye on the organizations and efforts he mentions:

• We just did a blog post last month on the Top 25 Most Dangerous Programming Errors.
• Port80 has looked to the Web Application Security Consortium’s Threat Classification project as a baseline for our WAF development - it is, as he says, a very clear and cogent taxonomy of Web App threats.
• We even have to pat ourselves on the back a bit for being mentioned as a resource under the WASC Fingerprinting threat description - Port80 has been preaching the gospel of Server Anonymization since before ServerMask was even born in 2002.

I’d be interested to hear whether you think we’re on the right track, and what headaches you’re having trying to deal with all the varying WAF and security options. And as always, please let us know how we can make your job easier.

Cheers!

Jenny @ Port80

{follow us on Twitter @port80software.com}

Today is the second annual Data Privacy Day, designed to raise awareness and generate discussion about data privacy practices and rights. The goal is to help users, especially teens, understand what info should really keep to themselves online.

The event is mainly focused on educating consumers, but it got me thinking. Part of our mission with ServerMask is to tell everyone with a server, “Hey, you don’t have to broadcast information about you server if you don’t want to!”

Really, your headers (and cookies, and platform details) are your business. Why give anyone more data than they need? If someone wants to cause trouble, every little clue could help. ServerMask is a really easy tool (click here for a screencast) that helps you put a little privacy screen around your server.

Cheers!

Jenny @ Port80

{follow us on Twitter @port80software.com}

GateHouse alleges that Boston.com thwarted electronic security measures intended to prevent it from scraping GateHouse’s online content. The NYT Co. argues that the references are clearly fair use, as only the headline and first line of content appear, with a link to the original article on Gatehouse’s sites.

Whether or not you think it’s unusual that a site owner wants to prevent reputable links to their content (after all, lots of firms are paying big money to SEO consultants for link building), there’s no doubt that Gatehouse should be able to dictate exactly how they want their content used.

If only there were a cheap and easy way to control exactly who can link to your content! A rules-based tool letting you grant or deny access on any number of characteristics, such as IP address, referring URL, country or geographic location or even the existence of cookie. A tool that let you redirect deep links to your site to your ad-rich homepage instead. I’m sure Gatehouse would find that particularly useful.

Call off the guard dogs (and the lawyers!) -  our LinkDeny access control software does just that for anyone running IIS servers. From keeping internal resources internal, protecting copyrighted images, and keeping content “borrowers” from piggybacking on your bandwidth, LinkDeny does it all. (And it’s much cheaper than a lawsuit!)

Let us know if we can help safeguard your content.

Cheers!

Jenny @ Port80

{follow us on Twitter @port80software.com}

The result is much discussion about preventative coding (the state of New York has already drafted requirements for future government web applications based on this list). The idea is to get software developers to certify in writing that their code is free of the errors mentioned in the list, SANS said.

While everyone in InfoSec is thrilled with any improvement in security-minded programming, those of us on the ground know this isn’t the end of the story. It helps, but a layered security approach is still key. Even top-notch coding from here on out won’t eliminate the need for a good Web Application Firewall:

• First, there’s a time gap between finding a hole and writing the fix — In a recent blog post, Jeremiah Grossman summarized some advice in a great informIT article: “Use WAFs to quickly reduce the immediate exposure (time-to-fix), then fix the root cause (the code) as time and budget allow.”
• Second, there’s always the problem with legacy code – even the best of best practices can’t use time travel to go back and fix problems in legacy code; practically, what you face is a choice between costly refactoring of that code under the new policies or using a WAF.
• Third, it’s hard to control what you don’t write yourself – a lot of Web apps use and depend on 3rd party components and while you can certainly beef up your due diligence to try to make sure the vendor uses best practices (which is the point of the report), ultimately you rely on trust — and ‘trust but verify’ is better than ‘trust and hope for the best’
• Finally, some vulnerabilities might require solutions at the architecture layer rather than in implementation.  This is clear if you look at the “prevention and mitigations” sections of  the Top 25 list. The most diligent coders can’t overcome fundamental application vulnerabilities by, say, writing really secure classes and functions. In cases like this, even refactoring existing code under a regime of best practices might not be enough; you might be looking at starting from scratch.

All of these scenerios are common enough to be proof that WAF’s aren’t going anywhere anytime soon. That said, hopefully these new changes will mean insecure legacy code will eventually find itself extinct.

Cheers,

Jenny @ Port80 Software

PS - If you’re in the market for a good WAF, you know where to turn… Port80’s ServerDefender

Check them out at www.port80software.com/screencasts

Cheers!

Jenny @ Port80

You can read about the context of Content Negotiation for XHTML and HTML here but unfortunately she doesn’t actually explain how to accomplish this practically. On Apache it is pretty straightforward: you can use mod_negotiation. However for those running IIS there was no implementation at all for this task. This is the main reason we designed pageXchanger a number of years ago; it also addresses dealing with PNG images in some browsers and not others.

What Tina proposes is to properly serve XHTML; pageXchanger is the solution for IIS. If you want to serve content dictated by device (when the device sends appropriate HTTP Accept headers), pageXchanger can do that. If you want to serve content by language it can do that too.

Unfortunately the market has yet to directly discover the decade-old power of generalized content negotiation (even though our compression products actually require this concept as well). Instead most of those who try pageXchanger seem to mainly use the product to build clean URLs, which is a bit like using a screwdriver to hammer nails – it works, but doesn’t take advantage the best features of the product.

Many search engine “experts” seem to think a clean URL is relevant to indexability and high rank. Don’t get us wrong, we like clean URLs more than the next person; we wrote a paper on the very topic. The problem is that determining causality or correlation between query strings and rank is weak at best. It would appear in reality the variability and date information on the page are the more predictive factors, rather than the query string.

Very dynamic pages (regardless of query string or not) aren’t going to be favored by Google or any other bot if they change too rapidly. This makes perfect sense; there’s no motivation to index an object that will change in a few moments. While the query string is one clue to non-indexable dynamism, it is certainly not the most obvious; there are a number of headers that show this regardless of URL style. Forget cleaning all tell-tale signs of dynamic pages; page changes are easily exposed by a simple double fetch by the search bot. Maybe you are doing black hat SEO content cloaking or just simply making the mistake of believing dynamic pages can be well-indexed. Regardless of the appropriateness of SEO motivations - clean your URLs anyway. That’s just a good idea; even people like Tim Berners Lee think so. You know that guy — the one who invented the Web?

That said, while we certainly aren’t trying to tell people not to use pageXchanger to clean up their urls, it isn’t really designed for that. You’ll get the most impact from pageXchanger using it for true content negotiation, allowing you to transparently select and serve language, image, and other content based on the user’s browser or device type. Simply put, content negotiation is a great technology for the diversity of the Web, as one page really doesn’t fit all!

Cheers,
Port80