Why are websites suddenly requiring referer headers (and failing silently)???

[referermadness]

I've been noticing a very quirky trend lately and I'm baffled by it. In the past month or two, I've begun to notice sites breaking without a referer header.

 

As background: you'll of course remember the archaic days where referer headers were misused to do a whole bunch of things from feature detection to some misguided appearance of security. There are still some legacy sites that depend on it, but for the most part refer headers have been relegated to ###### device detection.

 

Imagine my surprise when not one, but three modern websites are suddenly breaking without a referer.

 

1. Codepen: pen previews and full page views just break. But editor view works perfectly.

 

2. Twitter: basically every interactive function breaks. If you try to tweet, retweet, favourite, etc. you get a generic no-descriptive error. If you try to update a setting, it just flat out refuses (403).

 

3. Imgur: It just can't upload anything and eventually gives up.

 

All three are modern websites. Codepen was already broken since I started using it so I'm not sure if it was always like that, but Twitter and Imgur used to work perfectly fine with no referer. In fact I had just noticed Imgur breaking.

 

Furthermore, all of them only generate non-descriptive error messages, if at all, which do not identify the problem at all. It took a lot of trial and error for me to figure it out the first two times, now I try referer headers as one of the first things. But wait! There's more! All it takes to un-bork them is to send a generic referer that's the root of the host (i.e. twitter.com, codepen.io, imgur.com). You don't even need to use actual URLs with directory paths!

 

One website, I can chalk it up to ###### code. But three, major, modern websites - especially when they used to work - is a huge head scratcher.

 

Has anybody else noticed this trend or know wtf is going on?

[pqt]

It sort of makes sense, but is a little strange for sure. My initial thought is that it's processing the actions to a different subdomain and then sending back the response. But that really depends how their internal systems are handling it. Analytics are everything to Twitter so I can understand that one because tracking what favourite or retweet came from where would be displayed in the analytics dashboard for advertisers and marketers.

 

Imgur and CodePen I am not entirely sure but perhaps for validation purposes? (purely a guess).

 

Edit: I tweeted this thread to Codepen so we can see what they say.

[primexx]

I don't get why they should be failing for any reason. If it's used for analytics it should never affect functionality. And if they're using it for any kind of security or access control then they're unbelievably misguided, and should not be doing it in the first place...

[Simon Veteran]

Well, if they want the data, they're probably intentionally making it difficult for people who don't want to give it to them. If it becomes a pain to browse the web without sending referrer headers, people might turn it off.

 

It might be a naive belief. But I think they also feel that the number of people disabling it is small enough that they can squash it early on. Last thing they want is the customer getting control, like what happened with adblockers. It is also their API's, and if they choose to treat requests without that header as malformed, then requests without that header are malformed to their API.

 

In each case, though, they should fail descriptively.

[The_Decryptor Veteran]

The referer header is so useless that I think anybody actually relying on it just doesn't know better. The only semi-reliable use for it is hotlinking prevention, and even then it's implemented wrong half the time (Sites requiring one that matches the domain, so bare image loads fail)