News

December 2, 2021updated 30 Sep 2022 10:48am

Left-wing bias or SEO supremacy? Inside Mail Online’s struggle to compete with The Guardian on Google

Mail Online may be one of the biggest English language news websites in the world – but it believes “woke” executives at Google systematically downgrade it in search.

Insiders at The Guardian, also a top ten global news site, meanwhile believe the issues the Mail has with visibility on Google are down to shortcomings on the site itself rather than search engine bias.

It is at least common ground that The Guardian does much better on search than Mail Online, with a visibility index of 357 versus 58, according to Sistrix which measures the presence of URLs in Google search results (see a full ranking shared by Sistrix below).

With Google every news website’s most important referrer of traffic, this issue is much more than a row between two ideologically opposed publishers. It is an issue at the heart of claims that Google abuses its position as the monopoly online search engine in the UK and the US.

Press Gazette spoke to both sides and a broad range of independent search engine experts to find out whether the search giant really does have it in for Mail Online…

Ofcom: Algorithms pose risk to plurality

The topic of search engine bias re-emerged recently after Ofcom published all the submissions to its future of media plurality consultation, which heard forthright views on Google from both Mail Online publisher DMG Media and Guardian Media Group.

Ofcom has since agreed that algorithms look likely to pose a risk to media plurality in the UK.

DMG Media told Ofcom that readers place “great faith in Google”, but added: “Unless they are students of search visibility, they have no idea that when they search for news Google invariably takes them to two left-leaning news sources, the Guardian and BBC.”

Mail Online questioned if there was a political bias against it and accused Google of punishing publishers selling adverts outside Google’s ad exchange (Google has denied doing this).

Guardian Media Group, on the other hand, believes the Google search rankings are based on “more objective – if still relatively uncertain – factors” in comparison to Facebook’s news feed algorithm. (According to Newswhip, Mail Online regularly sits second or third in the rankings of publishers on Facebook).

GMG acknowledged it is “not precisely clear” how any single factor affects what appears on a search engine results page (SERPs) but said, citing an SEO guide from optinmonster.com, that relevant factors include having a secure and accessible website, domain age, URL and authority, user experience, page speed, mobile friendliness, and technical SEO.

The Guardian’s director of public policy Matt Rogerson ran his own tests using Google’s free-to-use Lighthouse tool in the Chrome browser’s incognito mode last autumn and again in September this year (although it should be noted that Mail Online disputes the validity of his testing).

Rogerson said he found that Mail Online fell short on factors including performance, best practices, SEO, speed index, and total blocking time and reasoned this would explain why the website does worse in search rankings than The Guardian, which did better on every measure.

6/ I ran them again in Sept ‘21 adding the @Telegraph.

Green is good, red is bad.

We know that Google cites these factors as significant in how sites rank in search, so these data are objective reasons why other one publisher might fare better in search than another. pic.twitter.com/a6dRshYuh5

— Matt Rogerson (@MattRogerson) November 8, 2021

Mail Online argued that it rates positively for all the ranking factors listed on Optinmonster.com and that Lighthouse tests can yield many different results depending on CPU, memory and local internet speed interference.

‘Quite wrong’ to make ‘arbitrary decisions’

A Mail Online spokesperson told Press Gazette: “If, as the Guardian argues, Mail Online’s page speed (and that of other sites like the Sun and the Mirror) is such a problem why, when all sources of traffic are considered, do the public prefer visiting these websites to the Guardian?” They referred to the latest Pamco data which puts the tabloids squarely ahead of The Guardian.

“In truth, according to Google’s own page experience scores, Mail Online’s pages are ranked ‘good’ - the highest rating, and the same as the Guardian,” they added. “Yet despite this closeness of scoring, Google’s search ranking invariably favours the Guardian.”

Earlier this year Google rolled out a page experience update on mobile, with the desktop update expected in the new year. It was the first time Google has ever given advance warning to publishers of an algorithm change and for the first time it clearly stated the quality of experience as a page loads would be a ranking factor for search.

The Mail Online spokesperson said: “The truth is that the Guardian and Mail Online have largely equivalent performance scores but vastly different search visibility.

“The fact that Google can and does manipulate search ranking to suit its own purposes was amply demonstrated by its core algorithm change in June 2019, when Mail Online’s search visibility was reduced by 50% overnight, while other websites gained – the Mail’s visibility was then equally abruptly restored three months later, after we protested to Google at the highest level.

“It is quite wrong that an American-owned commercial monopoly is able to make arbitrary decisions about what news is presented to the British public and what is not. We continue to urge the Government to address this problem through regulation of algorithms by the Digital Markets Unit – in which Ofcom is an important stakeholder - as soon as possible.”

Mail Online also disputed The Guardian's take that it invests less in "softer" news topics and therefore ranks lower for them.

The Mail spokesperson said: "The Guardian makes much of the fact that it performs poorly in search for subjects like showbiz gossip and royal news – hardly surprising given that it rarely covers them. Even so, when the Guardian did cover a recent major royal event – the death of Prince Philip – Google promptly gave it top position in ranking for related search terms, despite its lack of authority in royal news.

"Conversely, Mail Online runs extensive content on political issues such as Brexit and Boris Johnson – in fact many attributed the electoral success of both to the Mail’s coverage. Yet, as the DMG Media submission showed, its share of Google search for those terms is close to zero."

A Guardian News and Media spokesperson told Press Gazette: "The Guardian invests heavily in digital to ensure our site is as good as it can be technically so millions of readers can access first-rate original Guardian journalism.

"We have no oversight of algorithmic decisions and have argued for much more transparency on this, especially around key updates. But claims of political or subjective bias need to be weighed against the hundreds of complex technical factors that we believe impact on search, as numerous SEO experts have also confirmed."

The Guardian also noted that Google has publicly stated its algorithm tries to prioritise original reporting (though not always successfully, according to concerns shared with Ofcom by The Sun). But Rogerson’s analysis showed that almost 60% of Brexit articles he found in Mail Online’s search archive carried an agency byline compared to 12% with a named byline (although 29% was marked unknown).

Heavy use of agency bylines is thought to be a factor that can see search performance downgraded. However Press Gazette understands Mail Online objects to this argument on the grounds that many of its wire stories are delisted from search and it only promotes its best journalism.

Rogerson said: “We can’t be sure what impact this has on DM’s position in search, but that volume of agency copy is significant. Certainly higher than we see on our site. It is possible that these two factors, rather than supposed subjective bias, are as relevant to their performance.”

In its Ofcom submission, Guardian Media Group said: “The promotion of original fact-based journalism by a native digital publisher over and above journalism produced by an incumbent, should be seen as a sign that competition is working, not as an example of subjective bias against an incumbent.

“It would be a poor outcome for consumers in any sector, if SERP were calibrated to promote products of a lower standard or poorer quality simply due to the incumbency or political power of the publisher of that content.”

SEO experts give their verdict

So, what do independent SEO experts make of the issue? Several told Press Gazette that they saw a tangible difference between Mail Online and The Guardian on technical performance.

Luke Budka, director of digital PR at Definition, said Mail Online was beating The Guardian on Core Web Vitals, essentially signals that Google considers important relating to how fast a website loads and becomes usable.

He added that although they both passed the Core Web Vitals Assessment based on sitewide metrics for 28 days in October to November (which he said The Sun had failed) the Mail had won in a review of two specific URLs: its coverage of an "insulate Britain zealot" compared to The Guardian’s coverage of the cuts to the HS2 rail project.

However he said The Guardian did “pip” Mail Online with its implementation of structured data, the code that can be added to webpage HTML to let Google know what it is crawling.

Budka said: “The implementation of structured data is important. Google states that if you have duplicate pages for the same content, you should place the same structured data on all page duplicates, not just on the canonical page i.e. if you have an AMP version of an article then the structured data should match the main version. The Daily Mail does not do this – take the ‘insulate Britain zealot’ article - on its AMP page it lists one author in the structured data, on its main page it lists all three.

“It also lacks profile pages for its authors which is important. Author profiles are probably used by Google to assess the ‘expertise, authority and trust’ of a site. The Guardian however uses the same structured data in both its AMP and non-AMP stories. It also lists an author profile page,” Budka said, adding that The Sun was “bringing up the rear again” by marking multiple stories in its structured data meaning Google may struggle to know which one to use in its Top Stories carousel.

Maryum Sheikh, a B2B SEO expert and marketing executive at The Digital Voice, told Press Gazette that Google “prioritises the pages who offer the best experience with relevant and useful content for their users” and on this basis it was “clear that there is no discrimination when it comes to ranking SERPs”.

“The algorithm chooses the closest search result which passes hundreds of ranking factors and signals,” she said.

“With that in mind and as an unbiased user and an SEO expert, the reason for The Guardian's higher ranking and The Mail Online’s lower has more to do with the website performance and user experience, in which The Guardian wins on both instances.”

'No exact science at play'

Nick Boyle, SEO strategy director for Bolton SEO agency The Audit Lab, told Press Gazette: “With respect, it feels as though the publications are struggling to see the wood for the trees here. They’re focusing on granular scores as a defining ranking metric, whereas the truth of the matter likely lies within the wider depths of how Google processes, understands and assesses content - particularly content trustworthiness and quality.”

He found that both websites “perform well at a glance” on Google’s page insights report, with The Guardian passing the Core Web Vitals assessment on both desktop and mobile in a 17 November test and Mail Online passing on desktop.

“This isn't the be-all and end-all, however,” he said. “Looking at search visibility (using third-party tools), we can see that both publications have experienced a drop in their respective SEO visibility scores, with The Guardian seemingly falling further from a greater height than The Mail Online. The Mirror and The Express have also seen declines during what has been quite a turbulent year involving several key Google updates.

“There is no exact science at play here - just signals and the cumulative impact of those (many, many) signals.”

Boyle suggested publishers read Google's Search Quality Guidelines and “how Google fights disinformation” document to help them be “first to the post” on a wider range of topics.

He added that both Mail Online and The Guardian “can certainly improve their respective outputs to naturally grow their relevance to specific 'hot topics' - but the solution certainly isn't as simple as using a site performance score to dictate success. There are in fact many factors/signals at play here”.

Guido Ampollini, who has been in the search industry for almost 15 years at companies such as Google and Expedia and now runs the GA Agency digital agency in London, shared a table from SEO tool Moz showing how close the domain ratings of some of the top online newspapers are:

British online newspapers ranked by domain authority. Source: Moz via Guido Ampollini

He suggested that BBC News and Guardian may therefore be given an edge in the SERPs because of their “positive reputation and history of high-quality original reporting”, something that is taken into account by Google.

But, he added: “The slight edge awarded to the BBC and Guardian, one point above the others, might explain the different treatment received by Google. The problem is that Google treatment is part of how DA is calculated, as stated by Moz, hence the number might be different as a result of how Google treats the website instead of actually explaining why Google believes the BBC and Guardian are better news sources than the Daily Mail or The Sun.”

Ampollini also shared another table using PageSpeed Insights showing The Mirror and The Sun were the only websites falling short on Core Web Vitals over 28 days in October and November.

The Sun and Mirror are the only websites falling short of Google’s Core Web Vitals tests. Source: PageSpeed Insights via Guido Ampollini

He said: “A mix of technical and reputation factors might ultimately favour some websites over others, and the most likely way of catching up is focusing on marrying top-quality original journalism with a smooth user experience, even though it is easier said than done.”

David Soffer, co-founder of both TechRound and Tudor Lodge Digital, was more critical of both Mail Online and The Guardian regarding their performance on Google.

He said: “Both websites, although claiming to be very strong from a technical and SEO point of view are, in fact, not good performers at all.”

He found that both websites performed “badly” on Google’s page speed analysis, with Mail Online performing better on mobile (56%) than The Guardian (15%) but worse on desktop (49%) than The Guardian (58%).

Soffer said Mail Online’s homepage URL was “messy, badly optimised and incredibly bad for SEO” and that he would advise clients to change and update it to modern standards.

He said Mail Online failed on Google’s core web vitals test which shows how pages perform based on real-world usage data, and that a 14-second “time to interactive”, meaning the time taken from start to finish to load the page, was “obscene”.

He added that as Mail Online runs a lot of ads, there is a huge amount of JavaScript every page has to load, adding significant load time which is bad for SEO. The site also extensively uses meta keywords which are no longer relevant for SEO and therefore are “clogging up their site source code unnecessarily”, he said.

Meanwhile Soffer said The Guardian does pass the Core Web Vitals test set by Google, runs significantly less JavaScript and third-party code for advertising enabling faster load times, has cleaner URL structures, makes better use of Schema Mark Up code giving increased data and context about pages to Google, and makes less use of meta keywords potentially reflecting a more modern approach.

Despite this, he said “both websites from a speed and pure performance perspective, are pretty horrendous”.

Soffer went on to explain why The Guardian may rank higher for politics stories with showbiz left for Mail Online: “Interestingly, and pertinent to both sites is that when searching ‘politics’ via Google, the Guardian shows up throughout Google News and indeed organic search, whereas when I search ‘showbiz’ the same applies to the Mail Online,” he said.

“Very important to realise is that Google does categorise sites in a sense. For example, a website will generally (apart from the likes of the BBC) have to have a fairly or clearly defined niche in which they sit and for which they target and rank for keywords. The same will apply when it comes to news websites.

“When Google’s algorithm crawls and reads the Mail Online, it will undoubtedly see and read the stories in the sidebar which are tabloid-esque, whereas a lot of what is easily found on the Guardian is politics-esque. This also means that there will be a huge amount of internal links (a fundamental SEO factor) pointing to showbiz stories on the Mail and politics in the Guardian. Moreover, this also means that the crawling of Google is guided strongly towards the internally linked stories. Each story in the sidebar on the Mail for example is an internal link and likewise for the politics stories on the Guardian.

“Although Google can manually penalise websites and do so from time to time, it is in all likelihood more a case that the Guardian are content-wise more angled towards politics whereas the Mail is towards showbiz. The Mail would be able to start angling themselves in a much stronger way towards the politics side of things but this would need a strategy when it comes to SEO to start angling Google’s algorithm towards the relevant factors and updates. This would be both technical as well as content related.”

'Like comparing rifles to vegan pyjamas'

Marc Swann, director of search at Glass Digital, told Press Gazette there are so many elements that contribute to SEO that simply looking at technical performance to explain visibility was “a bit misleading”.

He similarly saw Mail Online and The Guardian operating in different environments, with less than 1% of the same keywords.

“It would be like comparing a website that sells high-performance rifles vs. a website that sells vegan pyjamas – their competition is different, their content is different, the websites that link to them will be different, the keywords used to search for their products are different etc,” he said.

“These differences will mean each website will have differing levels of available search volume available to them and it could be easier or harder to maximise visibility due to competition.”

Several of the SEO experts shared some advice for publishers on how to improve their visibility.

Sheikh of The Digital Voice said: “User experience might be the new SEO for Google but it is certainly not the only essential one; the quality and relevance of content still wins.

“Google prioritises content which is most useful and relevant both to a user’s search query and a publisher’s niche. An overlooked factor is that Google tends to rank content that solves problems for users or satisfies the user search query, for which it uses hundreds of signals, AI, and algorithm checks. Ultimately, the content that wins is that which receives the highest engagement by humans.

“To ultimately rank higher, a publisher’s initial goal should be to satisfy their user and offer the best experience while interacting with their web page. That includes reducing the unnecessary scrolling for users to reach the heart of the story on a news page.”

'No smoking gun'

Ampollini, of GA Agency, concluded: “At the end of the day, scientifically explaining Google rankings is a hard task even for the best industry experts. While conspiracy theories are easy to advocate and official explanations are often vague, Google results can vary daily, hourly, and differ because of location, user history, and device.

“In the case of The Mail, Sun and other publishers versus BBC, Guardian and others, it seems like Google is deeming the latter portals as more authoritative, rather than left-leaning. Rather than bias, authority might be the issue, as well as page experience, even though correlation is different from causality.”

Swann of Glass Digital added: “There’s no smoking gun here, no single SEO related factor that is determining the Guardian should be more visible than the Daily Mail in search results.

“The real test would be for both teams to start from scratch, compete for the same set of keywords and see who comes out on top over a 12-month period.

“We can all get the popcorn out and watch.”

Regardless of what publishers do to update their SEO, Definition’s Budka pointed out they are no longer only competing with other publishers.

“They’re now competing with brand-side digital content teams too,” he said, giving the example of investment management company Hargreaves Landsdown which recently ranked fourth in Google News with its analysis of Tesco’s half-year figures – outranking The Times by cleverly using structured data. “Savvy flaks are now circumnavigating hacks altogether,” he said.