Show HN: Website Speed Checker in Bulk (using apify)

Website Speed Checker

Easily get Website Speed Data with Google Lighthouse performance metrics (Performance Score, FCP, LCP, TBT, CLS, Speed Index) from any webpage/URL, whether you're analyzing a single URL or thousands in bulk. Run the scraper for 1 to 100,000 pages, one time, on a schedule, or via API. This Actor uses Google Lighthouse to audit webpage performance for both desktop and mobile devices, providing detailed insights into Core Web Vitals and other key metrics to optimize user experience and site speed.

Use Cases

• Performance Monitoring: Track webpage speed and performance metrics over time for one URL or thousands to ensure optimal user experience.
• Site Optimization: Analyze Core Web Vitals (FCP, LCP, CLS) across single or multiple pages to identify and fix performance bottlenecks.
• Bulk Performance Audits: Conduct comprehensive speed audits for thousands of URLs in one go, ideal for large websites or enterprise-level analysis.
• Competitor Analysis: Benchmark the speed of competitor websites, from a single page to entire site structures, to inform strategic improvements.
• Development Testing: Validate performance improvements during development by auditing one URL or batches of staging/production URLs.
• Client Reporting: Generate detailed speed reports for a single page or thousands of URLs to demonstrate site performance to clients.

Important Note

Because we run the real chrome browser lighthouse in BOTH mobile and desktop, it takes on average 1 minute per URL. If you are running more than 40 at a time, please consider changing the timeout accordingly.

How to Use

Using the Apify UI

• Navigate to the Actor: Open the Website Speed Checker Actor in the Apify Console.

• Input URLs: Go to the "Input" tab in the Apify frontend UI. Enter the URLs you want to audit in the "URLs to audit" field. The field is prefilled with https://onescales.com/ for testing. For a single URL, enter one (e.g., https://example.com). For bulk analysis, add multiple URLs (e.g., https://example.com, https://example.com/about, https://example.com/contact), one per line or as a comma-separated list. You can audit up to 100,000 URLs in a single run.

• Adjust Settings (optional): Timeout: Set the maximum time (in seconds) for each page scan. Default is 1200 seconds (20 minutes). Minimum is 600 seconds. Memory: Specify the memory limit (in MB). Default is 4096 MB, with a minimum of 2048 MB. For bulk runs (e.g., thousands of URLs), increase memory to 8192 MB or higher in the run settings for optimal performance.

• Run the Actor: Click the "Start" button to begin auditing the URLs. Monitor progress in the Apify Console.

• View Results: Once the run is complete, go to the "Dataset" tab to view the extracted performance metrics. Results are displayed in a table format, with separate columns for desktop and mobile metrics.

• Using JSON Input For programmatic or API-based usage, provide input in JSON format. Below is an example JSON input configuration for auditing one or multiple URLs: { "urls": [ "https://example.com", "https://example.com/about", "https://example.com/contact" ] }

urls: An array of URLs to audit (required). Use one URL for a single page or multiple for bulk analysis (up to 100,000 URLs). timeout: Maximum time per page scan in seconds (optional, default: 300). memory: Memory limit in MB (optional, default: 4096).

To use this JSON:

In the Apify Console, go to the "Input" tab and select the "JSON" editor. Paste the JSON configuration. Click "Start" to run the Actor. Alternatively, use the Apify API to submit the JSON input programmatically. Refer to the Apify API documentation at https://x.ai/api for details.

Output Example

The Actor outputs data to a dataset with fields for both desktop and mobile performance metrics. Below is a sample output for a single URL: [ { "url": "https://www.rossstores.com/", "Desktop-statusCode": 200, "Desktop-performanceScore": 92, "Desktop-firstContentfulPaint": 1200, "Desktop-largestContentfulPaint": 1800, "Desktop-totalBlockingTime": 150, "Desktop-cumulativeLayoutShift": 0.02, "Desktop-speedIndex": 1400, "Mobile-statusCode": 200, "Mobile-performanceScore": 85, "Mobile-firstContentfulPaint": 1500, "Mobile-largestContentfulPaint": 2200, "Mobile-totalBlockingTime": 200, "Mobile-cumulativeLayoutShift": 0.03, "Mobile-speedIndex": 1600, "error": "" } ]

url: The audited webpage URL. Desktop/Mobile-statusCode: HTTP status code (e.g., 200 for success, null for failure). Desktop/Mobile-performanceScore: Lighthouse Performance Score (0–100). Desktop/Mobile-firstContentfulPaint: First Contentful Paint (ms). Desktop/Mobile-largestContentfulPaint: Largest Contentful Paint (ms). Desktop/Mobile-totalBlockingTime: Total Blocking Time (ms). Desktop/Mobile-cumulativeLayoutShift: Cumulative Layout Shift (unitless). Desktop/Mobile-speedIndex: Speed Index (ms). error: Error message if the audit fails (empty string if successful).

For bulk runs, the dataset will contain one such object per URL.

How to Export

Once the Actor completes its run, you can export the performance data from the Apify Console:

• Go to the Dataset: Navigate to the "Dataset" tab in the Apify Console for your Actor run.

• Choose Export Format: Select from formats like JSON, CSV, Excel, XML, or RSS. For most users, CSV or Excel is ideal for analyzing data in spreadsheets, while JSON is suitable for programmatic use.

• Download or Integrate: Click the export button to download the dataset in your chosen format. Alternatively, use the Apify API to retrieve the dataset programmatically. Refer to https://x.ai/api for API endpoint details.

• Bulk Data Handling: For large datasets (e.g., thousands of URLs), ensure you have sufficient storage and processing capacity when exporting to formats like Excel. Use the "Clean" option in the Apify Console to filter out failed audits (where error is non-empty) if needed.

Need Help

Have Questions or Need Additional Features? We’re here to support you! Whether you need help with setup, have questions about interpreting results, or want to request additional features for the Website Speed Checker, we’ve got you covered. Contact Us Just fill out the form at https://docs.google.com/forms/d/e/1FAIpQLSfsKyzZ3nRED7mML47I4LAfNh_mBwkuFMp1FgYYJ4AkDRgaRw/viewform?usp=dialog, and we’ll try our best to help as quickly as possible.