Fifteen years ago, the world wide web was the playground of boffins. Its design reflected the open ethos of those users: it had no central managers, no main menu and no investment in content – indeed, no business plan whatsoever.
Instead, its framers assumed that people would put their own material online, and users would then surf from one site to another, following links on the pages.
Then the first search engines sprang up, which sent digital robots crawling from one link to the next, copying everything they found. The idea was to index the entire web in one place by obsessively following every path from several starting points.
Soon you could search the web by entering a search term and finding all the pages that contained it. This shortcut rankled with some webmasters. Even though they"d chosen to put their data on the web for all to see, they felt far more exposed once any words they used within their pages could turn up as search results.