Proper Pixel Art – Convert Pixel-Art-Style Images into Usable Assets

4 months ago 4

Converts text-to-art generated pixel-art-style images from LLMs to true pixel resolution assets.
Extracts pixel art from screenshots.

git clone [email protected]:KennethJAllen/generative-pixel-art.git cd generative-pixel-art

Create Virtual Environment

Install uv if not already installed.
Sync environments
- uv sync

First, obtain a source image (e.g. generated pixel art with a transparent background or a screenshot of low‑quality art).

uv run ppa -i <input_path> -o <output_path> -c <num_colors> -p <pixel_size> [-t]

Flag Description

-i, --input <path>	Source image file in pixel-art-style
-o, --output <path>	Output path for result
-c, --colors <int>	Number of colors for output. May need to try a few different values (default 16)
-p, --pixel-size <int>	Size of each “pixel” in the output (default: 20)
-t, --transparent	Output with transparent background (default: off)

uv run ppa -i data/creatures/blob.png -o . -c 16 -p 20 -t

The algorithm is robust. It performs well for images that are already approximately alligned to a grid.

Here are a few examples. A mesh is computed, where each cell corresponds to one pixel.

Generated by GPT-4o. Note the noisy edges.

Screenshot from Google images of Pokemon asset.

Original image generated by GPT-4o.

Screenshot from Google Images of Stardew Valley asset.

This tool can also be used to convert real images to pixel art by first requesting a pixelated image of the original image from GPT-4o, then using the tool to get the true resolution of the iomage.
Consider this image of a mountain

Here are the results of first requesting a pixalated version of the mountain, then using the tool to get a true resolution pixel art version.

The result of pixel-art style images from LLMs are noisy, high resolution images with a non-uniform grid and random artifacts. Due to these issues, standard downsampling techniques do not work. How can we recover the pixel art with "true" resolution and colors?

The current approach to turning pixel art into useable assets for games are either

Use naive downsampling which does not give a result that is faithful to the original image.
Manually re-create the image in the approperiate resolution pixel by pixel.

The main algorithm solves these challenges. Here is a high level overview. We will apply it step by step on this example image of blob pixel art that was generated from GPT-4o.

Note that this image is high resolution and noisy.

Trim the edges of the image and zero out pixels with more than 50% alpha.
- This is to work around some issues with models such as GPT-4o not giving a perfectly transparent background.
Find edges of the pixel art using Canny edge detection.

Close small gaps in edges with a morphological closing.

Use Hough transform to get the coordinates of lines in the detected edges. Only keep lines that are close to vertical or horizontal giving some grid coordinates.

Find the grid spacing by filtering outliers and taking the median of the spacings, then complete the mesh. Cluster points that are closeby together.

Quantize the original image to a small number of colors.
- Note: The result is sensitive to the number of colors chosen.
- The parameter is not difficult to tune, but the script may need to be re-run if the colors don't look right.
- 8, 16, 32, or 64 typically works.
- TODO: Find a heuristic for choosing the number of colors.
In each cell specified by the mesh, choose the most common color in the cell as the color for the pixel. Recreate the original image with one pixel per cell.
- Result upsampled by a factor of $20 \times$