Open source tool to collect data for computer use agent models

A privacy-first application for collecting standardized computer use data to train AI agents.

A simple, user-friendly program that records your computer interactions—keyboard inputs, mouse movements, screen recordings, and audio—to help train and improve AI agents. Unlike proprietary solutions, all data stays on your machine until you explicitly decide to share it.

No coding required. Just install and run.

As AI agents become more capable of computer use tasks, they need high-quality training data. However, current data collection methods are:

• Fragmented: Everyone uses different formats and collection methods
• Privacy-invasive: Data often gets uploaded automatically to remote servers
• Closed-source: No transparency in what's being collected or how

This library solves these problems by providing:

• Standardized data format: Consistent, well-documented schemas for interoperability
• Local-first privacy: All data stored locally; you control what gets shared and when
• Full transparency: Open-source code you can audit and modify

• Easy to Use: Simple GUI or command-line interface—no coding needed
• Privacy by Design: Everything stored locally on your machine
• Standardized Format: Common data schema for cross-platform compatibility
• Selective Recording: Choose what to record (keyboard, mouse, screen, audio)
• Transparent: Fully open-source and auditable
• Your Data, Your Choice: Export and share on your own terms
• Lightweight: Minimal performance impact on your system
• User-Friendly Interface: Clean, intuitive design for everyone

• Operating System: macOS, Linux (Windows support coming soon)
• Python: 3.8 or higher (comes pre-installed on most systems)
• Disk Space: 100 MB for the app + storage for your recordings
• RAM: 200 MB minimum
• For MP4 video: ffmpeg (install with brew install ffmpeg)

Don't worry if you're not sure—the installer will check for you!

If you're on macOS Sequoia (15.x):

• All recording modules work! (keyboard, mouse, screen)
• Uses separate processes to avoid tkinter/pynput conflicts
• Just grant Input Monitoring + Screen Recording permissions in System Preferences

Note: Most Mac and Linux systems come with Python pre-installed. If you get an error, you may need to install Python first from python.org.

Option 1: Graphical Interface (Recommended)

This opens a simple window where you can:

• Click "Start Recording" to begin a session
• Choose what to record (keyboard, mouse, screen, audio)
• View your recorded sessions
• Export data when you're ready to share

Option 2: Command Line

1. Run computeruse-collect to open the app
2. Click "Start Recording"
3. Do your computer work as normal
4. Click "Stop Recording" when done
5. Your data is saved locally in ~/computer_use_data/

Default Storage Location:

On macOS/Linux, this expands to:

Folder Structure: Each recording session creates its own folder:

Accessing Your Files:

• Via GUI: Open the app → Click "View Sessions" → Select a session → Click "Export"
• Via Terminal:
  # Open the storage folder open ~/computer_use_data # List all sessions ls ~/computer_use_data

Changing Storage Location: You can change where files are stored:

1. Open the app
2. Click "Settings"
3. Find "Storage Path" and click "Browse..."
4. Select your preferred location

The app provides a clean, easy-to-understand interface:

All collected data follows a standardized JSON schema with timestamps, making it easy to:

• Replay sessions for debugging
• Train AI models with consistent inputs
• Share data across different platforms and research groups

Example data structure:

• Keyboard events (what keys are pressed)
• Mouse coordinates and clicks
• Screen recordings (optional, configurable quality)
• Audio input/output (optional, off by default)

• No automatic uploads to any server
• No personally identifiable information (unless you explicitly include it)
• No background collection when not in a recording session

When you're ready to share, you have full control:

Via GUI:

1. Open the app with computeruse-collect
2. Click "View Sessions" to review what you've recorded
3. Select sessions you want to share
4. Click "Export" to create a shareable file
5. Upload the exported file to wherever you choose (research project, AI company, etc.)

Via Command Line:

• Help Train AI: Contribute your computer interactions to improve AI agents
• Support Research: Help researchers understand how people work with computers
• Earn Rewards: Some AI companies may compensate users for quality data contributions
• Personal Productivity: Analyze your own work patterns
• Accessibility: Help improve technology for users with different abilities
• Open Science: Contribute to open-source AI development

Customize collection settings through the app or config file:

Via GUI Settings:

• Open the app and click "Settings"
• Adjust privacy, performance, and storage options
• Changes take effect immediately

Via Command Line:

Via Config File (~/.computeruse-collect/config.json):

Make sure Python's bin directory is in your PATH. Try:

If you see this error on macOS Sequoia, uninstall the mss library:

The app will automatically use native macOS APIs for screen capture instead.

The app needs permission to record your screen/keyboard. Go to: System Preferences → Security & Privacy → Privacy

Make sure you have Python 3.8+:

We'd love your help making this better! You don't need to be a developer:

• Report bugs: Found something broken? Let us know!
• Suggest features: Have ideas? Share them!
• Improve docs: Help make this README clearer
• Code contributions: Add features or fix bugs
• Translations: Help make this accessible worldwide

See CONTRIBUTING.md for guidelines.

This tool is designed for consensual data collection. Users should:

• Only record their own computer use
• Be aware when recording is active
• Review data before sharing
• Never record others without explicit consent
• Never use for surveillance or malicious purposes

MIT License - see LICENSE file for details.