As a long time data scientist and engineer, I've had to write a couple of quick and dirty scrapers and bots over the years using selenium and more recently playwright. I haven't really been tracking it, but I've also been reading about the crawl4ai project.
With the explosion of AI agents, I've been playing around with building agentic scrapers that can simply be given a prompt and a target site and are able to return structured data in a specified format. I've also been playing around with adding in steps that have a different model/step attempt to define the structured format dynamically.
However, as with most AI projects, the token consumption can scale pretty aggressively.
Has anyone else been working on similar projects? Would people realistically pay $0.025 to $0.03 per request?