Google’s Trust & Safety Team allegedly were in charge of monitoring a blacklist of “problematic” websites. The memo was leaked to The Daily Caller, who published excerpts on April 9th, 2019.
The memo stated…
“The deceptive_news domain blacklist is going to be used by many search features to filter problematic sites that violate the good neighbor and misrepresentation policies. […]
“The purpose of the blacklist will be to bar the sites from surfacing in any Search feature or news product. It will not cause a demotion in the organic search results or de-index them altogether,
We saw this in effect for various websites, including sites like News With Views and RedState, neither of which were entirely de-indexed but they were filtered and shadow banned from Google News, with their results being hidden from the public’s view. In fact, if you check Google News, none of the content appears if you simply search for “RedState.com”.
A recent archive of the search captured on April 11th, 2019 shows that none of the articles from RedState.com appear when you search for it in Google News using the term “RedState.com”.
However, if you use the term “site:RedState.com”, you’ll get an actual listing of the site’s Google News entries. You can view an archive of the listing to confirm that the site is approved for Google News, but the news is not visible to the general public.
This is called a shadow ban.
It’s still active to this very day against RedState.com.
As noted by the Daily Caller, the de-indexing reportedly did not remove targeted or blacklisted websites from the “ten blue links” portion of the search results but it did affect visibility for the Top News” carousel, the “Videos” section, the “Google News” section, and certain sidebar results.
The article paraphrases some of the memo and also adds a bit of context to some of the references, stating…
“That document reads that it was, ‘approved by [email protected], [email protected], [email protected] as of 8/13/2018.” Ben Gomes is Google’s head of search, who reports directly to CEO Sundar Pichai. Pandu Nayak is a Google Fellow, and Paul Haahr is a software engineer, whose bio on Google’s internal network Moma indicates that he is also involved in, ‘fringe ranking: not showing fake news, hate speech, conspiracy theories, or science/medical/history denial unless we’re sure that’s what the user wants.’
The memo also details what could impact a website based on the “Misrepresentation Policies” and “Good Neighbor Policy”.
It’s stated as follows…
“If your product shows any of the following, Misrep and GNP would apply to your PA.
“Shows content from users and news publishers (percieved 3P voice). Ex: UGC, News corpus, etc.
“Outputs single answers (perceived to come from the open web). Ex: Web answers, Video answers, etc.
“Shows content owned, licensed, or edited by Google (perceived to come directly from Google). Ex: Knowledge panels, News summaries, Oneboxes, Munin carousels, etc.”
As noted by The Daily Caller, during the judiciary hearing back in December of 2018, Google CEO Sunder Pichai stated that Google engineers and staff do not manipulate the search results. He claimed…
“This is working at scale, we don’t manually intervene on any particular search result.”
According to The Daily Caller, the list was last updated on December 3rd, 2018, as noted in the document.
In one section that they quote, the blacklist tool was only in use by a limited number of personnel at Google, and there were steps to being added to the blacklist, which included a watchlist. The document stated…
“The beginning of the workflow starts when a website is placed on a watchlist which is used for monitoring of sites to determine if they violate the Good Neighbor Policy. This watchlist is maintained and stored by Ares with access restricted to policy & enforcement specialists working on the Good Neighbor Policy. Access to the listing can also be shared at the discretion of pcounsel and legal investigations on a need to know basis to enforce or enrich the policy violations. The investigation of the watchlist is done in the tool Athena, the Ares manual review tool, and intakes signals from Search, Webspan, and Ares in order to complete reviews. … Once a domain is determined to be violating the misrepresentation policy or the Good Neighbor Policy, such patterns are then added to deceptive_news_blacklist_domains.txt by the Trust & Safety team.”
After publishing the news, The Daily Caller reached out for a comment from Google but didn’t receive a reply until after publication.
Google’s representative reiterated what Pichai said, stating…
“We do not manually determine the order of any search result, nor do our algorithms or policies attempt to make any judgement on the political leanings of a website. Our Google News inclusion policies are publicly available online. They provide guidelines on content and behaviors for matters like sponsored content, deceptive practices, and more. Sites that do not adhere to these policies are not eligible to appear on news surfaces or in information boxes in Search. These policies do not impact the way these sites appear in organic blue-link Google Search results.”
This is verifiably false.
One Angry Gamer was actually impacted by such a blacklist on two occasions. And for reference, One Angry Gamer has been an approved for Google News since 2016.
Nevertheless, the first occurrence happened shortly after June 28th, 2017 after we published an article about Anita Sarkeesian harassing Sargon of Akkad at the 2017 Vidcon event. The article ended up being featured in Google’s “Top Stories” news courasel when searching up the term “Anita Sarkeesian” or “VidCon” or “Sargon of Akkad”.
The site was promptly DDoS’d and then was never featured in the “Top Stories” carousel thereafter, until two years later.
The second occurrence was a lot more impactful, and it took place shortly after the date listed in the memo where the blacklist was allegedly employed on August 13th, 2018.
After August 18th, 2018, One Angry Gamer content no longer showed up in the “News” section on Google under any of the general search terms. The site also had a number of articles de-indexed or pushed into 60th or higher position in Google’s general search engine.
Since it was likely that Google would deny any wrongdoing, archives and web caches of the manipulated search results were made, and much like RedState.com, it was discovered that One Angry Gamer content no longer appeared in the Google News section under general searches. An archive from August 30th, 2018 verifies that searching for “One Angry Gamer” returned null results. The only way to have seen content from the site was to use the term “site:oneangrygamer.net” or filter the results by “Date”.
Multiple examples of the shadow bans, including other websites that were impacted by this measure, were outlined in an article published on August 29th, 2018.
This persisted for seven months.
After contacting Google support about the issue multiple times throughout that duration, we were told that Google does not manipulate the search results and that the issue was with the site. However, we never received a notice or any warning that we had violated any of the Google News rules. I asked multiple support staff at Google why the site was being shadow banned instead of it didn’t violate any of the rules? I never received a response.
However, around March 13th, 2019, the site was no longer shadow banned.
No reason was given, and no one from Google notified us why the site was removed from the shadow ban list.
One Angry Gamer began appearing again in the “Top News” carousel and now appears normally in the “Google News” section, as indicated with the most recent search results for “One Angry Gamer” in the news section.
Comparisons of the search results and shadow banning was posted on Twitter for posterity purposes.
After ending up in Google’s shadowban jail for covering Anita Sarkeesian harassing Sargon at Vidcon in 2017, we’re back in the Google News carousal for the first in two years.
We’re back, boys. pic.twitter.com/lnQ8Np4d74
— One Angry Gamer (@OneAngryGamerHD) March 14, 2019
*Whoops correction: 19% drop, not 25%.
— One Angry Gamer (@OneAngryGamerHD) March 14, 2019
As for the impact of the de-indexing, the site was only pulling in 25% of its traffic from Google by the end of the seven month run in the shadow ban gulag.
This wasn’t just based on internal numbers from the trackers, even Alexa.com noted that the site’s search traffic had dropped by a massive 19.00%.
So nearly one fifth of all traffic to the site had been wiped out over the course of seven months due to the shadow ban.
This means that either all of the captured data, images, search archives, and traffic monitors are wrong, or Google is lying.
As the old saying goes: the numbers don’t lie, people do.
(Main image courtesy of NetworkGreeks)