Show HN: Nightcrawler – A mitmproxy-based scanner to find low-hanging fruit

Version: 0.9.0

A mitmproxy addon for background passive analysis, crawling, and basic active scanning, designed as a security researcher's sidekick.

WARNING: BETA Stage - Use with caution, especially active scanning features

• Acts as an HTTP/HTTPS proxy.
• Performs passive analysis:
  • Security Headers (HSTS, CSP, XCTO, XFO, Referrer-Policy, Permissions-Policy, COOP, COEP, CORP, basic weakness checks).
  • Cookie Attributes (Secure, HttpOnly, SameSite).
  • JWT Detection & Decoding (in Headers and JSON responses).
  • Basic Info Disclosure checks (Comments, basic keyword context - Note: API/Key/Secret checks temporarily disabled).
• Crawls the target application to discover new endpoints.
• Runs basic active scans for low-hanging fruit:
  • Reflected XSS (basic reflection check).
  • SQL Injection (basic error/time-based checks).
  • Stored XSS (basic probe injection and revisit check).
• Configurable target scope, concurrency, payloads, and output via command-line options.
• Logs findings to console and optionally to a JSONL file.

You can install nightcrawler-mitm directly from PyPI using pip (once published):

It's recommended to install it in a virtual environment. For development/local testing:

• Navigate to project root directory (containing pyproject.toml)
• Activate your virtual environment (e.g., source .venv/bin/activate)

Once installed, a new command nightcrawler becomes available. This command wraps mitmdump, automatically loading the addon. You MUST specify the target scope using the --set nc_scope=... option.

You can pass any other valid mitmproxy arguments (like --ssl-insecure, -p, -v) AND Nightcrawler-specific options using the --set name=value syntax.

1. Configure Browser/Client: Set proxy to 127.0.0.1:8080 (or specified port).

2. Install Mitmproxy CA Certificate: Visit http://mitm.it via proxy.

3. Run Nightcrawler:

   • Specify Target Scope (REQUIRED!): nightcrawler --set nc_scope=example.com

   • Common Options (Combine as needed): nightcrawler -p 8081 --set nc_scope=example.com nightcrawler --ssl-insecure --set nc_scope=internal-site.local nightcrawler -v --set nc_scope=example.com # Use -v or -vv for debug logs nightcrawler --set nc_max_concurrency=10 --set nc_scope=secure.com nightcrawler --set nc_sqli_payload_file=sqli.txt --set nc_output_file=findings.jsonl --set nc_scope=test.org

   • Show Nightcrawler & Mitmproxy version: nightcrawler --version

   • Show all Nightcrawler and Mitmproxy options (look for 'nc_' prefix): nightcrawler --options

   NOTE: If nc_scope is not set, Nightcrawler will run but remain idle.

4. Browse: Browse the target application(s). Findings appear in the terminal and optionally in the specified JSONL file.

Nightcrawler configuration follows this precedence:

1. Command-line --set options (highest precedence)
2. Values in configuration file
3. Built-in defaults (lowest precedence)

Configuration File:

• By default, Nightcrawler looks for a YAML configuration file at:
  • ~/.config/nightcrawler-mitm/config.yaml (on Linux/macOS, standard)
  • %APPDATA%/nightcrawler-mitm/config.yaml (on Windows, needs check)
  • Fallback: ~/.nightcrawler-mitm/config.yaml (if XDG path not found/writable)
• You can specify a different configuration file path using the --nc-config option when running Nightcrawler (passed via --set): nightcrawler --set nc_config=/path/to/my_config.yaml ...
• The configuration file uses YAML format. Keys should match the addon option names (without the --set).

Example config.yaml:

You can always override defaults or config file values using --set. This takes the highest precedence.

To see all available ncoptions and their current effective values (after considering defaults, config file, and --set), run: nightcrawler --options | grep nc

• If you specify relative paths for nc_output_file or nc_output_html (either in the config file or via --set), Nightcrawler will attempt to save them relative to a default data directory:
  • Linux/macOS (XDG): ~/.local/share/nightcrawler-mitm/
  • Windows (approx): %LOCALAPPDATA%/nightcrawler-mitm/
• If you specify absolute paths (e.g., /tmp/report.html), they will be used directly.
• Nightcrawler will attempt to create these directories if they don't exist.

• Basic Active Scans: Scanners are basic, intended for low-hanging fruit. Cannot detect complex vulnerabilities. DO NOT rely solely on this tool.
• Stored XSS Detection: Basic implementation, may miss cases and have FPs.
• Info Disclosure: Content checks for keys/secrets are basic and currently disabled pending refactoring.
• Resource Usage: Tune --set nc_max_concurrency.
• False Positives/Negatives: Expected. Manual verification is required.

This project is licensed under the MIT License. See the LICENSE file for details.

Contributions welcome! See the GitHub repository: https://github.com/thesp0nge/nightcrawler-mitm