Show HN: A Comprehensive AI Data Quality Evaluation Tool

👋 join us on Discord and WeChat

• 2024/12/27: Project Initialization

Dingo is a data quality evaluation tool that helps you automatically detect data quality issues in your datasets. Dingo provides a variety of built-in rules and model evaluation methods, and also supports custom evaluation methods. Dingo supports commonly used text datasets and multimodal datasets, including pre-training datasets, fine-tuning datasets, and evaluation datasets. In addition, Dingo supports multiple usage methods, including local CLI and SDK, making it easy to integrate into various evaluation platforms, such as OpenCompass.

Example config_gpt.json:

After evaluation (with save_data=True), a frontend page will be automatically generated. To manually start the frontend:

Where output_directory contains the evaluation results with a summary.json file.

Try Dingo on our online demo: (Hugging Face)🤗

Try Dingo in local:

Experience Dingo interactively with Google Colab notebook:

Dingo includes an experimental Model Context Protocol (MCP) server. For details on running the server and integrating it with clients like Cursor, please see the dedicated documentation:

English · 简体中文 · 日本語

To help you get started quickly with Dingo MCP, we've created a video walkthrough:

mcp_demo.mp4

This video demonstrates step-by-step how to use Dingo MCP server with Cursor.

Dingo classifies data quality issues into 7 dimensions of Quality Metrics. Each dimension can be evaluated using both rule-based methods and LLM-based prompts:

Quality Metric Description Rule Examples LLM Prompt Examples

COMPLETENESS	Checks if data is incomplete or missing	RuleColonEnd, RuleContentNull	Evaluates if text abruptly ends with a colon or ellipsis, has mismatched parentheses, or missing critical components
EFFECTIVENESS	Checks if data is meaningful and properly formatted	RuleAbnormalChar, RuleHtmlEntity, RuleSpecialCharacter	Detects garbled text, words stuck together without spaces, and text lacking proper punctuation
FLUENCY	Checks if text is grammatically correct and reads naturally	RuleAbnormalNumber, RuleNoPunc, RuleWordStuck	Identifies excessively long words, text fragments without punctuation, or content with chaotic reading order
RELEVANCE	Detects irrelevant content within the data	RuleHeadWord variants for different languages	Examines for irrelevant information like citation details, headers/footers, entity markers, HTML tags
SECURITY	Identifies sensitive information or value conflicts	RuleIDCard, RuleUnsafeWords	Checks for personal information, and content related to gambling, pornography, political issues
SIMILARITY	Detects repetitive or highly similar content	RuleDocRepeat	Evaluates text for consecutive repeated content or multiple occurrences of special characters
UNDERSTANDABILITY	Assesses how easily data can be interpreted	RuleCapitalWords	Ensures LaTeX formulas and Markdown are correctly formatted, with proper segmentation and line breaks

Dingo provides several LLM-based assessment methods defined by prompts in the dingo/model/prompt directory. These prompts are registered using the prompt_register decorator and can be combined with LLM models for quality evaluation:

Prompt Type Metric Description

TEXT_QUALITY_V2, TEXT_QUALITY_V3	Various quality dimensions	Comprehensive text quality evaluation covering effectiveness, relevance, completeness, understandability, similarity, fluency, and security
QUALITY_BAD_EFFECTIVENESS	Effectiveness	Detects garbled text and anti-crawling content
QUALITY_BAD_SIMILARITY	Similarity	Identifies text repetition issues
WORD_STICK	Fluency	Checks for words stuck together without proper spacing
CODE_LIST_ISSUE	Completeness	Evaluates code blocks and list formatting issues
UNREAD_ISSUE	Effectiveness	Detects unreadable characters due to encoding issues

Prompt Type Metric Description

QUALITY_HONEST	Honesty	Evaluates if responses provide accurate information without fabrication or deception
QUALITY_HELPFUL	Helpfulness	Assesses if responses address questions directly and follow instructions appropriately
QUALITY_HARMLESS	Harmlessness	Checks if responses avoid harmful content, discriminatory language, and dangerous assistance

Prompt Type Metric Description

TEXT_QUALITY_KAOTI	Exam question quality	Specialized assessment for evaluating the quality of exam questions, focusing on formula rendering, table formatting, paragraph structure, and answer formatting
Html_Abstract	HTML extraction quality	Compares different methods of extracting Markdown from HTML, evaluating completeness, formatting accuracy, and semantic coherence
DATAMAN_ASSESSMENT	Data Quality & Domain	Evaluates pre-training data quality using the DataMan methodology (14 standards, 15 domains). Assigns a score (0/1), domain type, quality status, and reason.

Prompt Type Metric Description

CLASSIFY_TOPIC	Topic Categorization	Classifies text into categories like language processing, writing, code, mathematics, role-play, or knowledge Q&A
CLASSIFY_QR	Image Classification	Identifies images as CAPTCHA, QR code, or normal images

Prompt Type Metric Description

IMAGE_RELEVANCE	Image Relevance	Evaluates if an image matches reference image in terms of face count, feature details, and visual elements

To use these assessment prompts in your evaluations, specify them in your configuration:

You can customize these prompts to focus on specific quality dimensions or to adapt to particular domain requirements. When combined with appropriate LLM models, these prompts enable comprehensive evaluation of data quality across multiple dimensions.

Dingo provides pre-configured rule groups for different types of datasets:

Group Use Case Example Rules

default	General text quality	RuleColonEnd, RuleContentNull, RuleDocRepeat, etc.
sft	Fine-tuning datasets	Rules from default plus RuleLineStartWithBulletpoint
pretrain	Pre-training datasets	Comprehensive set of 20+ rules including RuleAlphaWords, RuleCapitalWords, etc.

To use a specific rule group:

• Data Sources: Local files, Hugging Face datasets, S3 storage
• Data Types: Pre-training, fine-tuning, and evaluation datasets
• Data Modalities: Text and image

• Built-in Rules: 20+ general heuristic evaluation rules
• LLM Integration: OpenAI, Kimi, and local models (e.g., Llama3)
• Custom Rules: Easily extend with your own rules and models
• Security Evaluation: Perspective API integration

• Interfaces: CLI and SDK options
• Integration: Easy integration with other platforms
• Execution Engines: Local and Spark

• Quality Metrics: 7-dimensional quality assessment
• Traceability: Detailed reports for anomaly tracking

If the built-in rules don't meet your requirements, you can create custom ones:

See more examples in:

• Register Rules
• Register Prompts
• Register Models

After evaluation, Dingo generates:

1. Summary Report (summary.json): Overall metrics and scores
2. Detailed Reports: Specific issues for each rule violation

Report Description:

1. score: num_good / total
2. type_ratio: The count of type / total, such as: QUALITY_BAD_COMPLETENESS / total
3. name_ratio: The count of name / total, such as: QUALITY_BAD_COMPLETENESS-RuleColonEnd / total

Example summary:

• WanJuanSiLu: A High-Quality Open-Source Webtext Dataset for Low-Resource Languages Uses Dingo for comprehensive data quality assessment of multilingual web data

• DataMan Methodology: DataMan: Data Manager for Pre-training Large Language Models Dingo implements the DataMan methodology for pre-training data quality assessment
• RedPajama-Data-v2: RedPajama-Data Dingo implements parts of the RedPajama-Data-v2 methodology for web text quality assessment and filtering

• Richer graphic and text evaluation indicators
• Audio and video data modality evaluation
• Small model evaluation (fasttext, Qurating)
• Data diversity evaluation

The current built-in detection rules and model methods focus on common data quality problems. For specialized evaluation needs, we recommend customizing detection rules.

• RedPajama-Data
• mlflow

We appreciate all the contributors for their efforts to improve and enhance Dingo. Please refer to the Contribution Guide for guidance on contributing to the project.

This project uses the Apache 2.0 Open Source License.

This project uses fasttext for some functionality including language detection. fasttext is licensed under the MIT License, which is compatible with our Apache 2.0 license and provides flexibility for various usage scenarios.

If you find this project useful, please consider citing our tool: