About a fifth of all News UK traffic in the last six months was non-human, chief technology officer Tom Jackson has revealed.
But he said bot attacks were even higher before it deployed new bot protection technology which has made its sites “less of a target”.
Around a third of that amount ended up getting blocked because it was judged to be “malicious bot traffic”, Jackson said.
“So 6% of our traffic we’re currently blocking,” he told a panel about the “AI traffic dilemma” at Press Gazette’s Future of Media Technology Conference.
Jackson said the publisher of The Times and The Sun started with an AI block list, letting everything else through.
Parent company News Corp is following what chief executive Robert Thomson has described as a “woo and sue” strategy, meaning “we’d like you to be our partner, but if you’re stealing our stuff we are going to sue you”. News Corp has AI licensing deals with OpenAI and Meta but is suing Perplexity.
Jackson said News UK has now moved to an “allow list” so only things they chose to specify were allowed to scrape their sites. But he said this was still “not really nuanced and sophisticated enough”.
News UK is now using bot protection software Netacea which Jackson said “basically looks for bot traffic and decides which bots it’s going to block and which isn’t”.
This is how News UK learned the scale of its non-human visitors.
Jackson said the process of blocking 6% of traffic was “probably not” watertight “but it’s pretty good, I think”.
He added: “The benefit of working with a third party like that is that they go off and hang out in the shady corners of the internet and the dark web in the chatrooms, hearing what the latest is on malicious bot scraping, and then refining their product to help combat that. So I would say that’s been pretty effective for us.”
Brian Alvey, chief technology officer at WordPress VIP, said the scale of News UK’s non-human traffic “sounds so low – it feels like there’s so much more scraping and activity going on”.
Jackson responded: “Actually, when we first switched on Netacea, those numbers were significantly higher, and they’ve come down a lot. I think because we’ve implemented that, we’re seen as less of a target.”
Jackson also revealed News UK has done work with AI marketing platform Profound to look at how often its brands appear in AI experiences for certain types of content compared to their competitors.
He said: “It told us that nothing’s really changed in terms of: where we would expect to be strong in presence, we’re still strong. Where we would expect others to be strong, they’re also still strong.”
He said AI has not led to a “cliff edge” of a traffic drop at News UK but it has “sharpened the focus that we already have, to be honest, on building a more direct relationship with our audience. That’s that’s a healthy thing anyway.”
In August, the UK audience for The Times was up 9% year on year (to 12.9 million) but The Sun was down 6% (to 21.2 million). Both grew page views, by 1% and 9% respectively. The Times is focusing on its app, which is used by 70% of subscribers, while The Sun introduced its Sun Club paid memberships at the start of last year.
Jackson added that publishers “do have to be a bit careful about reading too much into early trends” but shared recent analysis of where News UK content is getting impressions in AI engines.
He said Google impressions (including via its AI experiences) “seem to be skewing more to evergreen type content, you know, travel content from The Times or health content from The Sun for example, whereas ChatGPT seems to be skewing more to live and timely content.
“But I think it’d be a bit rash to suddenly jump on that and build an entire strategy around that when, as we all know, in three months’ time that could look very different,” adding that it was best to optimise for both AI experiences instead of prioritising one.
Daily Mail testing commercial opportunity for LLM visibility
Carly Steven, director of SEO and editorial e-commerce at the Daily Mail, said the publisher is now moving from “monitoring to enforcing now” with AI bots and scrapers.
The Daily Mail does not have a licensing deal with any major AI players although it was an early investor in Prorata, which promises to share revenue with publishers by the proportion of their content used to create an AI-generated answer.
Steven said the Daily Mail’s strategy has been to block AI bots by default but that it is now testing “a bit more of a nuanced approach to that”.
“Where the opportunity naturally presents itself, I think for us right now, is in the commercial space,” she said, meaning where the Daily Mail can help its brand partners be more visible in LLMs like ChatGPT.
But she said it was important that Daily Mail still has “control – so it’s on our terms. It isn’t an all-or-nothing.”
Steven said early tests have been “very positive” and that commercial partners are coming to the Daily Mail to discuss this opportunity as they know readers are “looking for information and answers from sources that they trust” on those platforms.
Steven added: “This isn’t a defensive play. This is actually opening up new conversations around revenue, and that’s really exciting and something that we obviously want to explore more.”
The Daily Mail’s UK audience was down 17% in August compared to the year before, to 16.7 million, but page views were up 5% to 403.5 million. Steven said algorithm changes were having more of an impact on traffic than AI Overviews and other AI products.
Alvey from WordPress VIP warned that AI licensing deals are harder to do after the first time.
“What do you do when you have a 200-year-old magazine or newspaper with maybe 20 years of content online, and they pay you a certain amount? Next year you only have one new year of content – what are they paying for that? Is that sustainable?”
Alvey also flagged the growth in the market for vendors helping publishers charge AI bots “a fraction of a penny” for content each visit.
“There’s a reason it’s called micropayments, it’s not a lot of money, and hopefully they can evolve from what they’re doing today into some kind of solution. I hope one of those companies wins. I hope one of them actually is the answer for all this.”
The Atlantic chief executive Nicholas Thompson told the Channels podcast this month his newsbrand is using AI start-up Parallel’s Index platform to bring in AI micropayments for “content that is uniquely valuable, hard to replace, or used in high-value agent work”.
Thompson said he had realised “the vast majority of traffic coming out of my computer was no longer human. It was my agents running requests. I was like ‘God, we need to build an Atlantic business model for that.’ That led us to Parallel.”
Alvey emphasised there are still occasions where traffic spikes are reassuringly human and show people’s appetite for breaking news, pointing to one of his publisher client’s stories on the death of singer Dolly Parton last month.
The site, which he did not name, normally does around three million visitors per hour during the night and five million during the day. But its Dolly Parton story had 99 million visitors in the first hour and 40 million the hour after that.
“We classify all of the traffic in our dashboards by colour,” he said. “Purple is humans… it was all purple. It was all people. There were no bots in there. Bots weren’t going nuts to find out about Dolly Parton. They knew about it an hour later… It was very nice to see a spike of human activity.”
Email pged@pressgazette.co.uk to point out mistakes, provide story tips or send in a letter for publication on our "Letters Page" blog
