Showcase video:
showcase-pg-db-optimize.mp4Integrated application using Python, Grafana, Vector and Loki to store explain analyze query results and display them for analyzation in a simple dashboard.
The motivation is to have many PostgreSQL explain analyze queries and databases and execute everything in one run, then analyze the results interactively on the dashboard.
Inspired by pev2. Please check out this amazing tool.
The python program uses the queries and databases saved to execute the explain queries. Results get logged using Python logging and vector picks up the logs from the docker container running the Python program. Vector sends it to Loki for storage, and Grafana queries Loki to display data in the dashboards.
Keep in mind that this application only measures the elapsed time of SQL statements as they execute within the database engine itself. It does not account for any of the additional latencies and overhead you’ll encounter in a real‐world setting, including but not limited to:
-
Network latency between your application server and the database host
-
Client‐side processing, such as building and serializing the query or parsing and deserializing the result set
-
Driver and ORM overhead, including marshalling parameters, mapping rows to objects, and any client‐side caching
-
I/O contention on the database server (disk reads/writes, log flushing, buffer cache misses)
-
Resource throttling or CPU scheduling enforced by container orchestrators or hypervisors
Because these factors can often dominate end‐to‐end response times, be sure to profile your full application stack—including the network path and client code—if you need an accurate measure of total latency.
Please have Docker Engine or Docker desktop installed.
Databases running in other docker containers running locally must be on the same Docker network as the program. To do this, each container running a PostgreSQL database must be started using the same docker network.
Create network
Setup external network with PostgreSQL databaes in docker-compose
Set env var NETWORK in .env
Do not include & in any database or query fields. The system doesn't sanitize &, which causes hard-to-diagnose errors or silent failures.
Only EXPLAIN ANALYZE queries in JSON format are supported. Always format SQL statements as:
- Navigate to the project directory:
Option 1: Using uv
Option 2: Using pip:
After running either option: Add databases and queries, then save them.
After adding queries and databases, you can run the rest of the application.
The start.sh will delete any data from previous run and execute the queries, begin the pipeline and make the dashboard ready.
Fresh start (deletes previous data)
Preserve Existing Data
Note: Query execution time depends on complexity. The dashboard will be available after processing.
http://localhost:3000/a/ivarehaugland-explaindbdashboard-app/home