Open source image generation with style codes (–sref))

Abstract

Innovative visual stylization is a cornerstone of artistic creation, while generating and representing novel styles remains a persistent challenge.Existing generative methods often rely on style images, lengthy textual descriptions, or parameter-efficient fine-tuning (PEFT) to guide models in generating images with specific styles. However, these methods struggle to create novel styles and require complex representations to convey stylistic information.In this paper, we affirm that a style is worth one numerical code by introducing the novel task, code-to-style image generation, which produces images with novel, consistent visual styles conditioned solely on a style code.To date, this field has only been explored by the industry(e.g., Midjourney), with no open-source research from the academic community.Specifically, we train a discrete style codebook to extract style embeddings from reference images. These embeddings then condition a T2I-DM to generate an image that aligns with the reference style.Specifically, we first train a discrete style codebook to extract style representations from reference images.Then, we train a text-to-image diffusion model (T2I-DM) conditioned on the output of the codebook, enabling it to generate images with specific style.Using the style codebook, we encode a large set of style images into indices and train an autoregressive model on them to model their distribution, which acts as the style generator.During inference, a numerical code deterministically samples a novel sequence of indices from the transformer. This sequence is then condition the diffusion process, generating style images.Unlike existing methods, our approach offers unparalleled simplicity and diversity, unlocking a vast space of reproducible styles from minimal input.Extensive experiments validate that SeeTyle effectively turns a single code into a powerful style controller, demonstrating a style is worth one code.