.png)
3 months ago
5
So you want to use AI and/or machine learning to make art...
This post is meant to be a guide to the tools/methods/software/etc. for making art with AI. I say “making art with AI” (rather than “AI art”) because this is meant to be a broader resource, both for artists who are interested in incorporating non-synthesis machine learning methods into their creative practice (like object detection, head/body/hand tracking, pose estimation, etc.), as well as anyone looking to create rich media (images, video, audio, etc.) with generative AI: what has come to be classified as “AI art” for better or worse.†
This post is organized in a top-down manner, with the most user friendly resources and tools first, and slowly gets more technical as well as media specific (video, audio, 3D, etc.) as it goes on. The intention with this is that you will skim/read until you find the point that is more low level (technically) than you care to get into*. And within each section, I’ve roughly arranged them in descending order of my preference (which not necessarily the objective best/fastest/cheapest/etc.).
For a broader resource covering more general creative coding tools, that you might want to use with what is listed here, check out terkelg/awesome-creative-coding or thatcreativecode.page.
This first group of tools and resources are both the fastest way (usually) to create what you want/need and the most user friendly. What you gain in usability and efficiency, you usually have to give up in cost and flexibility. Most of these have either a free tier or a set amount of free credits however (probably to get you hooked).
Products/Platforms
- Midjourney: one of the more artistic/aesthetic/cinematic models.
- DALLE 3 (OpenAI)
- ⭐️ Runway - Generative video and AI powered video editor.
- Adobe Firefly
- Recraft: Generative AI for design/stock/products.
- Topaz Labs: Professional-grade photo and video editing powered by AI (upscaling, touch up, restoration, denoise, etc.).
- ClipDrop: regenerative AI tools for image and photo transformation/touch up.
APIs
These APIs are usually much more general (and developer centric) than the platforms listed above, which are designed like more specific creative tools/software. The benefit to these APIs, is that they are much easier to incorporate into a larger project, if you need to do more than just generate assets. And they function more so like “hosted models,” rather than end-to-end software/tools, so they often have the SoTA models as they come out and give you slightly more flexibility/specificity in the parameters you can use too.
Visual Programming and Self Hosted GUIs
These tools are usually what I recommend folks get started with (if you don’t have any more specific wants/needs). These are all open source projects that have vibrant communities that contribute new methods/models somewhat frequently, are quite powerful/flexible, and are reasonably easy to use once you get them setup on your computer locally. And they’re free!
The downside to these is that they require a somewhat powerful computer/GPU, and they can be a little bit frustrating to setup (though they have all gotten much more user friendly than they used to be).
- ⭐️ ComfyUI: powerful open source, visual programming language (node-based tool) that incorporates many of the latest and greatest open source models/techniques, in a much more accessible way (compared to programming libraries). The community is also very good, and there are a plethora of user contributed nodes to usually do anything you could want.
- Stable Diffusion Web UI: definitely the easiest of the open source tools to get started with, but not as flexible as ComfyUI or programming libraries like Hugging Face Diffusers.
- Ollama: Software to download and run LLMs locally (with optional SDK libraries).
- LM Studio: similar in scope to Ollama.
Programming Libraries
At the bottom of our abstraction pyramid, we have straight programming libraries. These are by far the most flexible, powerful, etc. and are where new models/research/etc. usually comes out first (though the above GUI tools incorporate new methods reasonably quickly). As such, if you want to use the absolute state-of-the-art or cutting edge, you unfortunately not only need to have a somewhat powerful computer/GPU/infrastructure, but you also need to be able to program (for now) and setup the libraries (that can have messy dependencies).
- ⭐️ Hugging Face Diffusers and Transformers
- LangChain
- llama.cpp
General (non-AI) Creative Coding Tools/Frameworks
These creative coding frameworks don’t provide any (or much) AI/ML functionality, but they can usually be integrated with other deep learning and/or AI libraries (or vice versa). And what they don’t provide in machine learning capabilities, they more than make up for it in creative coding capability.
- ⭐️ Processing (Java) and p5.js (Javascript)
- openFrameworks (C++)
- Cinder (C++)
- nannou (Rust)
- Blender: Open source 3D computer graphics software that can be scripted with Python.
- Godot: Open source game engine that can be scripted with C# or extended with C/C++.
- Love2D (Lua): Open source 2D game engine in Lua. As such, it can be integrated with C/C++ programs pretty seamlessly either using the C API or LuaJIT’s FFI library
Creative ML Frameworks
These are non-generative machine learning libraries that can be integrated with a larger art project, and provide functionality like machine perception (object and pose detection, face tracking, etc.)
General Purpose Deep Learning
Learning
I won’t say much on the learning resources below, mainly because they all have decent descriptions/outlines/schedules that provide much more context than I ever could in a pity sentence. Instead, I grouped them into categories by topic (i.e. deep learning) as well as medium (course vs. video vs. book).
My recommendation here is to browse the outlines/TOCs to look for topics (or terms) you know you want to learn more about. And if you don’t know what/where to start, the starred resources are pretty good introductions to each category.
Courses
General Deep Learning
- Practical Deep Learning for Coders (fast.ai)
- ⭐️ Deep Learning for Computer Vision (UMich)
- Introduction to Deep Learning (CMU)
- Deep Learning for Computer Vision (Stanford CS231n)
- Natural Language Processing with Deep Learning (Stanford CS224n)
Deep Generative Modeling
- ⭐️ Learning-Based Image Synthesis (CMU)
- Deep Generative Models (Stanford)
- Deep Unsupervised Learning (UC Berkeley)
- Differentiable Inference and Generative Models (Toronto)
- Learning Discrete Latent Structure (Toronto)
Creative Coding and New Media
- ⭐️ Deep Learning for Art, Aesthetics, and Creativity (MIT)
- Machine Learning for the Web (ITP/NYU)
- Art and Machine Learning (CMU)
- New Media Installation: Art that Learns (CMU)
- Introduction to Computational Media (ITP/NYU)
Explainer Videos
- ⭐️ The AI that creates any picture you want, explained (Vox)
- I Created a Neural Network and Tried Teaching it to Recognize Doodles (Sebastian Lague)
- Neural Network Series (3Blue1Brown)
- Beginner’s Guide to Machine Learning in JavaScript (Coding Train)
- Two Minute Papers
Books
- ⭐️ Dive into Deep Learning (Zhang, Lipton, Li, and Smola)
- Deep Learning (Goodfellow, Bengio, and Courville)
- Computer Vision: Algorithms and Applications (Szeliski)
- Procedural Content Generation in Games (Shaker, Togelius, and Nelson)
People
And lastly, I want to mention some people and places doing interesting things at the intersection of art, ML, and design. Both for inspiration, but also for some exemplar examples of “AI art” out there.
- Sarah Friend
- Lauren McCarthy
- Anna Ridler
- Tom White
- Ivona Tau
- Margaret Murphy
- Emi Kusano
- Memo Akten
- Neural Bricolage (helena sarin)
- Sofia Crespo
- Philipp Schmitt
- Maya Man
- Trevor Paglen
- Sasha Stiles
- Mario Klingemann
- Tega Brain
- Mimi Onuoha
- Allison Parrish
- Caroline Sinders
- Anika Meier
- Robbie Barrat
- Kyle McDonald
- Golan Levin
Places
- STUDIO for Creative Inquiry
- ITP @ NYU
- Gray Area Foundation for the Arts
- Stability AI (Eleuther, LAION, et al.)
- Goldsmiths @ University of London
- UCLA Design Media Arts
- Berkeley Center for New Media
- Google Artists and Machine Intelligence
- Google Creative Lab
- The Lab at the Google Cultural Institute
- Sony CSL (Tokyo and Paris)
Postscript
I’ve tried to be somewhat specific in my curation of these resources, as to not provide an overwhelming list of links. If you do want an overwhelming list of links however, I put up an older (slightly outdated) resource on Github, or there are these other similar-in-spirit resources:
- Machine Learning for Art
- Tools and Resources for AI Art (pharmapsychotic) - Big list of Google Colab notebooks for generative text-to-image techniques as well as general tools and resources.
- Awesome Generative Deep Art - A curated list of Generative Deep Art / Generative AI projects, tools, artworks, and models
- ChristosChristofidis/awesome-deep-learning
comments/questions/concerns happen over on ✌️ Github
