Why the AI revolution needs tollbooths

Eighteen months ago as the AI chatbot revolution was taking hold, Olivia Joslin and Toshit Panigrahi both realized something profound was happening to the way the internet worked. 

AI web crawlers had begun inundating news and information websites with thousands of requests a day compared to the handful they typically saw from search engines. Not only was the explosion in traffic ballooning hosting costs for these sites, the bots supplied zero traffic to them in return. Web traffic in exchange for permission to crawl has been one of the unseen foundations of the internet economy for a generation.

Joslin, who is 29, was seeing this happen from the perspective of an AI company doing the crawling, Fairmarkit,. Panigrahi, who is also 29, was seeing it as the head of ads for Toast, the restaurant point of sale company where they’d both worked early in their careers. 

“It felt like we were seeing two sides of the same problem,” Joslin said. It also seemed like there wasn’t yet a good solution. The web wasn’t set up to easily compensate news and information websites for this new structural shift, they said. 

So they created Tollbit. It’s literally an online tollbooth. You sign up. You decide how much, if anything, to charge AI bots to crawl your website. And the next time they show up to crawl they get routed to Tollbit’s subdomain and hit with a paywall. Publishers can choose different prices for crawls to generate article summaries and for displaying the article’s full text. And it allows publishers to exclude some website data entirely. 

“It felt like we were back in the Napster days for the music industry and that maybe we could supply a Spotify-like solution – a recurring revenue model for the publishing industry,” Joslin said. 

Since then – just 18 months – Tollbit, has become one of the most talked about new ventures in the tech/media startup community. More than 2000 publications now use Tollbit’s system including Time, Newsweek,  AdWeek and the Associated Press. That list also includes publications owned by Penske Media, like Rolling Stone; publications owned by Mansueto Ventures, like Inc and Fast Company; publications owned by Lee Enterprises, which includes almost 80 newspapers; and publications owned by Hearst, which include 27 magazines like Elle and 30 newspapers including the San Francisco Chronicle. 

Tollbit processed more than 15 million transactions in the first quarter of this year, up from 5 million in the fourth quarter 2024. That volume is likely to be even higher when second quarter volumes are tallied.  It’s grown just past 20 employees. And it’s raised $31 million in two rounds from Lightspeed Venture Partners; Jeff Dean, Google’s chief scientist and cofounder of Google Brain; and Bill Maris, who started Google Ventures and is now the founding partner of S32.

If you’re tempted to yawn in the face of a story about internet plumbing, stop yourself. Tollbit has lashed itself to one of the most disruptive developments online since the invention of Google itself. 

Google essentially invented the business of crawling in exchange for monetizable traffic a generation ago with Adwords. It remains the source of its dominance today. And it has been an essential fuel for the growth of the $16 trillion global internet economy.

Now, AI chatbots are massively disrupting this relationship, and the disruption is only accelerating. Not only are more and more searches going through AI chatbots that generate zero traffic for publishers. Google itself is now sending publishers less traffic. Instead, Google is increasingly choosing to use its own AI product Gemini to respond to queries as a way of competing with the chatbots. 

This is the biggest challenge to Google’s advertising money machine that the company has ever faced.  Google has the money and talent to adjust. But super successful companies stumble more than they succeed when faced with such an existential threat. 

The speed that this five alarm fire is sweeping through digital publishing is breathtaking. The AI revolution is effectively only 31 months old. But from May 2024 to February 2025 traffic to the 500 most visited US websites fell 15 percent according to Axios. Some news sites have gotten hit much harder.  A recent piece in the Wall Street Journal said that monthly search traffic to Business Insider is down 55 percent in the last three years and by 33 percent in just the last 18 months to 50 million. 

It drove Business Insider’s decision earlier this month to lay off 21 percent of its editorial staff. That same piece quoted Nicholas Thompson, the CEO of the Atlantic, saying that he ultimately expected referral traffic from search to disappear entirely. “Google is shifting from being a search engine to an answer engine,” he said.

Online tolls is such a new business that it’s too soon to predict how meaningful first mover advantage will be. But we’re about to find out how much it matters. Cloudflare, the $60 billion content delivery network and cybersecurity giant, is gearing up to launch its own online tollboth on July 1, according to someone who has seen a draft of the press release. 

That could quickly disintermediate Tollbit. But it could just as easily be the best thing to happen to the company.  The market for online tolls is only as big as the AI companies allow it to be. They’re the ones paying the tolls. And it should be no surprise that right now they are dragging their feet. Cloudflare has the leverage to force more of them to sign on to this concept. That would expand transaction volumes for all players, including Tollbit. 

The one thing Cloudflare’s entry into online tolls shouldn’t be is a surprise. CEO Matthew Prince has been talking about AIs traffic disruptions at least since January.  Two months ago he said during an interview at the Council on Foreign Relations that “AI is going to fundamentally change the business model of the web…. We sit in between 80 percent of the AI companies. And 20 to 30 percent of the web uses Cloudflare. Part of what we’re thinking about is (what we can do about) that.” 

By now you’re probably wondering why online tolls might be a business at all. Aren’t many publishers solving this problem by blocking all the AI bots? They are. What most don’t realize, Tollbit’s Panigrahi told me, is that technically these aren’t blocks as much as requests. And he said that more and more AI companies are ignoring those requests. He said 3.3 percent of crawl block requests were ignored at the end of 2024. At the end of March 2025, it was up to 13 percent. 

Ok. But Isn’t there a limit on the amount of training data the AI companies will eventually need? Sure, big publishers like News Corp have inked nine figure deals with AI companies for access. And yes, the dozens of publisher lawsuits pending against AI companies will also most likely lead to deals eventually. There’s also a market for middlemen like Scalepost.AI to aggregate smaller content libraries and to take a percentage of those deals. But won’t transaction volumes for training data ultimately start to shrink, not grow? 

What many are just beginning to understand, Panigrahi told me, is that at least for questions about current events,  training data is only part of what enables an AI chatbot  to answer your question. 

Training data enables the AI chatbot to understand the question. And it enables it to answer the question if it’s on a topic with an established and rarely updated corpus of information. Want to know  “How did the Manhattan Project get its name?”  or “How many instruments does Paul McCartney play?” The AI chatbot you use will use its  corpus of training data to answer the question. 

But for questions about current events such as “Give me the two most popular columns in favor and against the US attacking Iran”  no AI chatbot is up to date enough. To do that, they rely on Retrieval Augmented Generation (RAG) technology to do, in effect, a real time search. After you ask, it crawls the best websites, and instantaneously assembles an answer. 

This is the technology driving most of the explosion in crawls to publisher websites, Panigrahi said. And as AI agents become a bigger and bigger part of web traffic as both human and AI agent use of AI chatbots explode in volume, he said it’s easy to argue that the volume of tollable crawls will expand in kind. 

“When we started, the world was very fixated on training data. We went to every single newspaper and media executive and the first thing they told us was “What’s the benefit of a toll? I’m going to either sue or I’m going to go do my one off deal,” Panigrahi told me. Now, according to Tollbit’s just released quarterly “State of the Bots” report, RAG crawls exceed training data crawls by 2.5 times on Tollbit’s network. Panigrahi said that trend is going to continue until “the majority of the internet is going to be AI traffic, not human traffic. The web is not configured for that today. So that’s the real opportunity we are chasing.” 

A guestimate of how big this new space could be is still complicated because it depends on so many hard to predict variables like the growth rate of AI bots, the crawling prices they are prepared to pay and the percentage of that revenue a company like Tollbit can grab. And Joslin and Panigrahi won’t tell me what those numbers look like inside Tollbit right now.

But it’s not hard to imagine how Tollbit could be generating as much as $71 million a year for its publishing partners right now. And that’s just Tollbit’s suggested pricing model if bots just want content to summarize. Tollbit suggests using the much higher syndication rate for bots that want to display an entire article. The math works like this: Total daily AI bot traffic to Tollbit client websites seems to be doubling every quarter right now. It was about 2.75 million at the end of the fourth quarter 2024, and 6.5 million at the end of the first quarter 2025.  If it doubles again in the second quarter to 13 million, that’s $195,000 a day Tollbit could be generating for its customers at the CPM of a typical content website of, say $15. $195,000 times 365 days is $71 million. 

Campbell Brown, Meta’s former head of media partnerships, who was one of Tollbit’s earliest investors, said what attracted her to Joslin and Panigrahi is that they’d figured out a way to get between publishers and the big technology companies. She’d seen how fraught that relationship had become from her time at Meta. “This was the first time I’d seen anyone present a possible solution.” It had only been four months since she’d left Meta, but she signed on as a senior advisor and began introducing the two cofounders to all the news executives she’d met during her seven years at the social media giant. 

Brown said that to her it’s obvious that AI chatbots are going to drive a lot of news consumption in the future. “All you have to do is look at my teenagers or their friends. These chatbots, not the New York Times, are their first touch point with news and information. That’s going to create a huge amount of competition for real time information among the big AI companies. Why? Because if your AI chatbot can’t tell me who won the game last night, give me five articles about it and do it right every time, my kids are going to find the one that does.”  

The prospect of a new revenue stream like this is tantalizing for publishers. For those with long tenures in the business it also feels richly deserved. One of the biggest mistakes the industry made 25 years ago was not taking Google’s rise seriously enough and therefore not demanding more compensation for letting Google crawl their websites when Google was small enough to be pushed around. “Pointing this out is often the last thing we tell publishers when we meet with them,”Panigrahi said. “ Imagine knowing what you know now about Google back then. This is your opportunity to turn the clock back 25 years.’”