DeepWiki to MdBook Converter

Relevant source files

• README.md

Purpose and Scope

This document introduces the DeepWiki-to-mdBook Converter, a containerized system that extracts wiki documentation from DeepWiki.com and transforms it into searchable HTML documentation using mdBook. This page covers the system's purpose, core capabilities, and high-level architecture.

For detailed usage instructions, see Quick Start. For architecture details, see System Architecture. For configuration options, see Configuration Reference.

Sources: README.md:1-3

Problem Statement

DeepWiki.com provides AI-generated documentation for GitHub repositories as a web-based wiki. The system addresses the following limitations:

Sources: README.md:3-15

Core Capabilities

The system provides the following capabilities through environment variable configuration:

• Generic Repository Support : Works with any GitHub repository indexed by DeepWiki via REPO environment variable
• Auto-Detection : Extracts repository metadata from Git remotes when available
• Hierarchy Preservation : Maintains wiki page numbering and section structure
• Diagram Intelligence : Extracts ~461 total diagrams, matches ~48 with sufficient context using fuzzy matching
• Dual Output Modes : Full mdBook build or markdown-only extraction via MARKDOWN_ONLY flag
• No Authentication : Public HTTP scraping without API keys or credentials
• Containerized Deployment : Single Docker image with all dependencies

Sources: README.md:5-15 README.md:42-51

System Components

The system consists of three primary executable components coordinated by a shell orchestrator:

Main Components

graph TB User["docker run"] subgraph Container["deepwiki-scraper Container"] BuildDocs["build-docs.sh\n(Shell Orchestrator)"] Scraper["deepwiki-scraper.py\n(Python)"] MdBook["mdbook\n(Rust Binary)"] MermaidPlugin["mdbook-mermaid\n(Rust Binary)"] end subgraph External["External Systems"] DeepWiki["deepwiki.com\n(HTTP Scraping)"] GitHub["github.com\n(Edit Links)"] end subgraph Output["Output Directory"] MarkdownDir["markdown/\n(.md files)"] BookDir["book/\n(HTML site)"] ConfigFile["book.toml"] end User -->|Environment Variables| BuildDocs BuildDocs -->|Step 1: Execute| Scraper BuildDocs -->|Step 4: Execute| MdBook Scraper -->|HTTP GET| DeepWiki Scraper -->|Writes| MarkdownDir MdBook -->|Preprocessor| MermaidPlugin MdBook -->|Generates| BookDir BookDir -.->|Edit links| GitHub BuildDocs -->|Copies| ConfigFile style BuildDocs fill:#fff4e1 style Scraper fill:#e8f5e9 style MdBook fill:#f3e5f5

Sources: README.md:146-157 Diagram 1, Diagram 5

Processing Pipeline

The system executes a three-phase pipeline with conditional execution based on the MARKDOWN_ONLY environment variable:

Phase Details

stateDiagram-v2 [*] --> ParseEnvVars ParseEnvVars --> ExecuteScraper : build-docs.sh phase 1 state ExecuteScraper { [*] --> FetchHTML FetchHTML --> ConvertMarkdown : html2text ConvertMarkdown --> ExtractDiagrams : Regex on JS payload ExtractDiagrams --> FuzzyMatch : Progressive chunks FuzzyMatch --> WriteMarkdown : output/markdown/ WriteMarkdown --> [*] } ExecuteScraper --> CheckMode state CheckMode <<choice>> CheckMode --> GenerateBookToml : MARKDOWN_ONLY=false CheckMode --> CopyOutput : MARKDOWN_ONLY=true GenerateBookToml --> GenerateSummary : build-docs.sh phase 2 GenerateSummary --> ExecuteMdbook : build-docs.sh phase 3 state ExecuteMdbook { [*] --> InitBook InitBook --> CopyMarkdown : mdbook init CopyMarkdown --> InstallMermaid : mdbook-mermaid install InstallMermaid --> BuildHTML : mdbook build BuildHTML --> [*] : output/book/ } ExecuteMdbook --> CopyOutput CopyOutput --> [*]

Sources: README.md:121-145 Diagram 2

Input and Output

Input Requirements

Sources: README.md:42-51

Output Artifacts

Full Build Mode (MARKDOWN_ONLY=false or unset):

output/ ├── markdown/ │ ├── 1-overview.md │ ├── 2-quick-start.md │ ├── section-3/ │ │ ├── 3-1-workspace.md │ │ └── 3-2-parser.md │ └── ... ├── book/ │ ├── index.html │ ├── searchindex.json │ ├── mermaid.min.js │ └── ... └── book.toml

Markdown-Only Mode (MARKDOWN_ONLY=true):

output/ └── markdown/ ├── 1-overview.md ├── 2-quick-start.md └── ...

Sources: README.md:89-119

Technical Stack

The system combines multiple technology stacks in a single container using Docker multi-stage builds:

Runtime Dependencies

Build Architecture

The Dockerfile uses a two-stage build:

1. Stage 1 (rust:latest): Compiles mdbook and mdbook-mermaid binaries (~1.5 GB, discarded)
2. Stage 2 (python:3.12-slim): Copies binaries into Python runtime (~300-400 MB final)

Sources: README.md:146-157 Diagram 3

File System Interaction

The system interacts with three key filesystem locations:

Temporary Directory Workflow :

1. deepwiki-scraper.py writes initial markdown to /tmp/wiki_temp/
2. After diagram enhancement, files move atomically to /output/markdown/
3. build-docs.sh copies final HTML to /output/book/

This ensures no partial states exist in the output directory.

Sources: README.md:220-227 README.md136

Configuration Philosophy

The system operates on three configuration principles:

1. Environment-Driven : All customization via environment variables, no file editing required
2. Auto-Detection : Intelligent defaults from Git remotes (repository URL, author name)
3. Zero-Configuration : Minimal required inputs (REPO or auto-detect from current directory)

Minimal Example :

This single command triggers the complete extraction, transformation, and build pipeline.

For complete configuration options, see Configuration Reference. For deployment patterns, see Quick Start.

Sources: README.md:22-51 README.md:220-227