What sites has Google been asked to remove from its index?

Who says Microsoft doesn’t play a significant role in Search?  According to Google’s latest Transparency report Microsoft makes the greatest number of domain and URL removal requests to Google (over three times more than number 2 on the list).

Google says that, in the past year, Microsoft, and agencies working for the company, have requested over 2.5 million individual web addresses be removed from Google’s search index from a total of 23,463 websites (the most recent of which was made today).  Apparently Microsoft makes over eight removal requests a week.

In the past month alone Microsoft requested that over half a million URLs (543,378) be removed by Google; this is over three times more than the second largest organisation on the list- the British Phonographic Industry, which asked for 162,601 URLs to be removed.

The data was released as part of Google’s biannual Transparency report.  With the report Google releases information on the number and type of requests it has received asking for the removal of data from Google’s search index.

But this report is significantly different from previous releases – this is the first time that Google has released information on companies rather than just governments.

And the numbers are just amazing – in the last month alone Google received over 1.2 million URL removal requests targeting nearly 25,000 sites from 1,300 copyright owners.

Previously Google only let us see which countries requested that data be removed; the type of complaint (defamation, national security, copyright) as well as the reason for the removal request (e.g. court order, government request).

Companies
Most of the requests for content removal coming from companies such as Microsoft are against file sharing websites, such as The Pirate Bay.  Looking at the data, most of the complaints are against music sharing – the British Phonographic Industry is one of the largest complainants on the site (although they’re joined by Sony Music, Universal Music, Warner Music, EMI, the Recording Industry Association of America, Ministry of Sound, and even the Estate of Michael Jackson).  NBC Universal was also highly active in requesting file sharing sites and data be removed.

The data in this release covers 95% of copyright claims made to Google since July 2011 using the company’s web form and only covers Search – removal request made against YouTube and Blogger are not included (so the copyright claim by AVG that forced YouTube to block the classic Rick Roll video for 24 hourswon’t be included).  Perhaps not surprisingly Google reports that an increasing number of removal requests are being made.

Countries
As for countries Google’s report, which covers the period January to June 2011, shows that the US made the greatest number of requests to access user data (5,950 – 93% of which were complied with), India came second with 1,739 requests (70% of which were complied with). France was third and also the highest ranked European country that requested user data, making 1,312 requests (47% of which were complied with).  The UK was close behind with 1,279 requests, 63% were complied with.  Germany comes third, with 1,065 requests, of which 66% were complied with.

If our calculations are right, this means that the UK (with 806 successful requests) then Germany (with 703 successful requests) placed a greater number of successful requests for user data than France (617 successful requests).   Collectively European Union countries placed 5,712 requests for user data, placing the Union a close second behind the US (with 5,950 requests).

Google says that it is seeing an increasing number of requests for user data for criminal investigations each year which it says “isn’t surprising” as its user base grows.

“The number of requests we receive for user account information as part of criminal investigations has increased year after year. The increase isn’t surprising, since each year we offer more products and services, and we have a larger number of users.”

Transparency
Google has been placing a significant amount of effort in releasing this data; this time last year the company activated realtime updates to its country-level transparency reports and in June 2011 they added even more data to the reports providing more information on the type of requests and the individual Google services which were targeted.