Fighting for quality news media in the digital age.

  1. Tech Platforms
February 1, 2024updated 08 Mar 2024 11:48am

Politico embraces generative AI web crawlers with website redesigns

Politico product and design chief Max Leroy explains thinking behind new website design.

By Bron Maher

Politico has given its Europe, German and French websites a redesign which improves readability for the web crawlers that feed into generative AI large language models (LLMs).

The relaunched Politico.eu sites, which went live last week, have simpler homepages, with fewer stories, more white space and the French and German-language editions hosting less English-language content.

Max Leroy, Politico’s vice president for product and design, told Press Gazette that although third-party cookies are not something the publisher relies on, their forthcoming deprecation is an “accelerator” for the outlet’s first-party data strategy.

How Politico.eu’s redesign hopes to bring in more first-party data

Readers do not have to log in to read Politico articles (except for those that are part of the B2B subscription product, Politico Pro), which means besides its newsletters it has few ways to gather first-party data on readers.

“We’re trying to change that and to offer features that will be of interest to people that aren’t logged in,” Leroy said. “It has the dual benefit of being great for readers and also for us to collect first-party data and not rely on third-party cookies.”

This does not mean Politico is implementing a registration wall, however. Instead, the publication hopes to give readers more reason to hand over their data through features like a personalised website experience and one-click newsletter sign-ups.

“We have people who come to our site like 15 times a day, and then we have people who have never heard about Politico,” Leroy said. “At the moment, they have exactly the same experience, because we have a single homepage that has, like any [news] website, just a focus on top news.”

Although the new homepage keeps that focus, a “latest news” tab has been added in the revamp.

“If you come every few hours, you will know exactly what’s new, and if you’re logged in it will even tell you what’s new since your last visit,” Leroy said, adding that such features “will set the foundation for us if we decide to have a registration wall, but that will be a larger conversation with all of Politico”.

The redesign is visual as well as technical, but even the sleeker new appearance is aimed at keeping audiences on Politico’s own platforms.

A graphic displaying the ability to swap between language editions on the new Politico website.
A graphic displaying the ability to swap between language editions on the new Politico website. Picture: Politico

“Another trend we’re playing against is that a lot of browsers are offering a reader experience taking out everything,” Leroy said. “People might have ad blockers. There’s just a lot of things that people do to basically read the content elsewhere to have a nice experience.

“And I think they’re less likely to do this, and more likely to stay on our website, if we have an experience that is closer to this.”

He said Politico had done “a lot of work on cleaning the article page, taking out some of those distractions, putting some related articles as blocks within the content of the article – rather than separated blocks – and those blocks will be controlled by the newsroom so that they can make more relevant recommendations…

“A block of like: ‘here are five more articles that you could read’ – I don’t think they work for any publisher. In my experience they never did.”

From SEO to AIO?

Leroy said Politico was thinking about how, besides traditional search engines and social media, people will find its journalism in future.

Although some of that will be decided “at the Axel Springer level, like the deal with OpenAI”, for Leroy’s team he said planning for the future was a matter of how content will be organised on Politico’s website.

“Perfect mapping of sections, subsections, constant breadcrumbs everywhere so that it’s very clear for a future AI crawler or new Google crawlers that our content is organised this way”.

He hoped these signposts will mean “we start showing up not just in regular search result pages, but in those generated answers in AI-style, conversation-style answers”.

Anyone who’s had to optimise an article for Google will be familiar with the process of tweaking topic pages, keywords and headings to make a story appear higher in search. But how will generative AI optimisation, as it were, differ from search engine optimisation (SEO)?

“I don’t think it differs, I think it’s accelerated,” Leroy said. “With traditional SEO, what every SEO expert will tell you is you have to make sure that your content was almost for the actual bot itself.

“Like: is it in the sitemap? Do you have the author? Do you have the right category and section attached to it?

“But most of the content in the article could somewhat stay relatively neutral – there was less SEO optimisation to be done within the article.

“Whereas if you look at most AI models and the way they’re going to process pieces of an article, they’re really going to anchor on H2s, H3s [i.e. subheadings] – are they able to understand that ‘this is a developing paragraph of this main concept’?”

By actively trying to get scooped up into the LLMs that power generative AIs, Politico is going against the trend for news publishers, many of whom have outright blocked the web crawlers deployed by Google and OpenAI.

[Read more: Major news publishers block the bots as ChatGPT starts taking live news]

Leroy said that with the deployment of new Google answer formats like Search Generative Experience, “when they’re citing different sources – we want to be in those”.

Recent coverage of SEO has been distinctly critical, with The Verge for example describing its practitioners as “the people who ruined the internet” for allegedly reducing the variety and usefulness of Google search query results.

Asked whether he thought optimisation for AI would make news sites more or less user-friendly, Leroy said he thought it would be a mix.

“Some of it will be worse… I know people in other publications [who have been] telling me we have seen these articles [which are] almost lookalikes of our articles, just a few words different, and it’s basically most likely a prompt asking the AI to, instead of copy and paste, make it just different enough so that it doesn’t rank as an exact copy.”

But he said the shift would also be “a wake-up call for some of the more traditional formats. Like, 1,000 word [articles with] no H2, no intermediary step to help you understand where you’re at in the article…

“I talked about how bots can scan the article – but if you read like me, you have less time than stories that you can read per day. And so most of the time I scan by H2.

“And the truth is, people reading A to Z perfectly, no matter how long [the text] – I think that’s rare, and I think we have to think about our audience as not that much different from bots. There is a bit of scanning, their interest is piqued by something, and then they will dive deeper and then they might read what was actually before and after.

“I think the reading experience in the way we have set it, or some publishers have it, is going to evolve for the best.”

Lighter, single-language pages to drive better Google traffic at Politico.eu editions

Leroy said the previous set up for Politico’s French and German editions had punished the site’s performance on Google.

“For example if you go on a French page, it ha[d] content in French and English. This is the worst signal we can send to Google. Because we’re telling Google: we don’t really know who that page is for – or it’s for bilinguals living in either the UK or France.

“In terms of traffic, we see that it’s doing well. But I know by experience that we have no idea of the ceiling here – it could have so much more potential.”

He pointed to the Spanish web edition of The New York Times, which he said has a “lighter template” than the English-language homepage, showing fewer stories.

“We want to have the same approach. The Germany and France homepage will be lighter, but because they will be lighter in a single language, they will actually drive more traffic, because they will be registered in France and in Germany.”

Another way Politico has addressed Google performance is with an edition-specific “hot topics” tab, surfacing only trending stories relevant to, for example, Politico Germany or France.

Politico relaunched its (single-language) UK site last year and Leroy said that brand was “already seeing a lot more traffic than the French page”.

The general, multi-language Europe edition will continue to exist, Leroy said, “and this one will keep ranking extremely well for big moments that we have across Europe”.

But the current set-up, he said, had been “basically trying to do two jobs at once” and ending up “not being ranked by Google for any of those two jobs”.

Email pged@pressgazette.co.uk to point out mistakes, provide story tips or send in a letter for publication on our "Letters Page" blog

Websites in our network