Show HN: Digital Twin Proxy – Turn web browsing into memory for AI agents

3 months ago 1

Turn web browsing into personal memory for AI agents.

Digital Twin Proxy logs web traffic and uses a local large language model (LLM) to generate an analysis of your browsing patterns. It's designed for developers, researchers, and anyone interested in understanding their online activity through the lens of AI.

HTTP/S Traffic Logging: Captures all web requests made through the proxy.
AI-Powered Analysis: Uses a local LLM (via Ollama) to analyze traffic.
Flexible Operation Modes: Run in the background, log traffic continuously, or perform one-off analysis.
Customizable: Easily change the AI model, analysis interval, and other settings.

Context for Agentic Applications

The primary output of Digital Twin Proxy is a structured log of your web traffic, along with AI-generated analysis. This data can serve as a powerful source of real-time context for other agentic applications.

By providing an analysis of recent browsing history, you can engineer a more informed context window for other AI agents, enabling them to:

Personalize responses: An agent can tailor its behavior based on your current tasks and interests.
Anticipate needs: An agent can proactively offer assistance based on the websites you are visiting.
Improve tool usage: An agent can better understand the context of your work and select the right tools for the job.

This process of "context engineering" allows you to create a more powerful and personalized AI experience.

We will soon expose the context from your digital twin as an MCP server to support AI agents.

In-Browser Context Injection

To create a more interactive and personalized web experience, we are developing a feature to inject real-time context directly into your browser for any agentic AI app (ChatGPT, Perplexity, etc.) to access your digital twin.

The proxy operates by routing your browser's traffic through a local Squid instance. Here’s the data flow:

Browser → Squid Proxy (port 8888) → Internet ↓ Access Logs ↓ Digital Twin Proxy App → Ollama → Analysis

Traffic Interception: Your browser is configured to send all HTTP and HTTPS requests to the Digital Twin Proxy listener on port 8888.
Logging: The proxy, powered by Squid, logs every request's URL and host.
Analysis: The digital-twin-proxy application monitors these logs, sending them to a local LLM via the Ollama API to generate a human-readable analysis of browsing patterns.

Rust toolchain
Squid proxy
Ollama with a running model (e.g., ollama run llama3)

Clone the repository and build the project:

git clone https://github.com/kstonekuan/digital-twin-proxy.git cd digital-twin-proxy cargo build --release

The binary will be located at target/release/digital-twin-proxy.

Configure Your Browser: Set your browser's HTTP and HTTPS proxy to 127.0.0.1:8888.
Verify: Start the proxy in logging mode and visit a website.

# Terminal 1: Start the proxy ./target/release/digital-twin-proxy log # Terminal 2: Tail the logs tail -f ~/.local/share/digital-twin-proxy/log.ndjson

You should see JSON objects representing your web traffic.

Digital Twin Proxy has three main commands:

log: Start the proxy and only log traffic.
analyze: Perform a one-shot analysis of traffic logged since a given duration.
ambient: Run the proxy and periodically analyze traffic in the background.

Examples:

# Log traffic without analysis ./digital-twin-proxy log # Analyze traffic from the last hour ./digital-twin-proxy analyze --since 1h # Run in ambient mode, analyzing every 5 minutes ./digital-twin-proxy ambient --interval 300

This project uses rustfmt for formatting and clippy for linting.

# Format code cargo fmt # Run linter cargo clippy --all-targets --all-features # Build and run tests cargo build cargo test

Contributions are welcome! Please feel free to submit a pull request or open an issue.

This project is licensed under the MIT License. See the LICENSE file for details.

Read Entire Article