Show HN: Evolved.lua – An Evolved Entity Component System for Lua

Evolved ECS (Entity-Component-System) for Lua

evolved.lua is a fast and flexible ECS (Entity-Component-System) library for Lua. It is designed to be simple and easy to use, while providing all the features needed to create complex systems with blazing performance. Before we start exploring the library, let's take a look at the main advantages of using evolved.lua:

This library is designed to be fast. Many techniques are employed to achieve this. It uses an archetype-based approach to store entities and their components. Components are stored in contiguous arrays in a SoA (Structure of Arrays) manner, which allows for fast iteration and processing. Chunks are used to group entities with the same set of components together, enabling efficient filtering through queries. Additionally, all operations are designed to minimize GC (Garbage Collector) pressure and avoid unnecessary allocations. I have tried to take into account all the performance pitfalls of vanilla Lua and LuaJIT.

Not all the optimizations I want to implement are done yet, but I will be working on them. However, I can already say that the library is fast enough for most use cases.

I have tried to keep the API as simple and intuitive as possible. I also keep the number of functions under control. All the functions are self-explanatory and easy to use. After reading the Overview section, you should be able to use the library without any problems.

And yes, the library has some unusual concepts at its core, but once you get the hang of it, you will find it's very easy to use.

evolved.lua is not just about keeping components in entities. It's a full-fledged ECS library that allows you to create complex systems and processes. You can create queries with filters, use deferred operations, and batch operations. You can also create systems that process entities in a specific order. The library is designed to be flexible and extensible, so you can easily add your own features and functionality. Features like fragment hooks allow you to manage your components in a more flexible way, synchronizing them with external systems or libraries. The library also provides syntactic sugar like the entity builder for creating entities, fragments, and systems to make your life easier.

On the other hand, evolved.lua tries to be minimalistic and does not provide features that can be implemented outside the library. I'm trying to find a balance between minimalism and the number of possibilities, which forces me to make flexible decisions in the library's design. I hope you will find this balance acceptable.

• lua >= 5.1
• luajit >= 2.0

You can install evolved.lua using luarocks with the following command:

Or just clone the repository and copy the evolved.lua file to your project.

To start using evolved.lua, read the Overview section first. It will give you a basic understanding of how the library works and how to use it. After that, check the full-featured Example, which demonstrates complex usage of the library. Finally, refer to the Cheat Sheet for a quick reference of all the functions and classes provided by the library.

Enjoy!

The library is designed to be simple and highly performant. It uses an archetype-based approach to store entities and their components. This allows you to filter and process your entities very efficiently, especially when you have many of them.

If you are familiar with the ECS (Entity-Component-System) pattern, you will feel right at home. If not, I highly recommend reading about it first. Here is a good starting point: Entity Component System FAQ.

Let's get started!

An identifier is a packed 40-bit integer. The first 20 bits represent the index, and the last 20 bits represent the version. To create a new identifier, use the evolved.id function.

The count parameter is optional and defaults to 1. The function returns one or more identifiers depending on the count parameter. The maximum number of alive identifiers is 2^20-1 (1048575). After that, the function will throw an error: | evolved.lua | id index overflow.

Identifiers can be recycled. When an identifier is no longer needed, use the evolved.destroy function to destroy it. This will free the identifier for reuse.

The evolved.destroy function takes one or more identifiers as arguments. Destroyed identifiers will be added to the recycler free list. It is safe to call evolved.destroy on identifiers that are not alive; the function will simply ignore them.

After destroying an identifier, it can be reused by calling the evolved.id function again. The new identifier will have the same index as the destroyed one, but a different version. The version is incremented each time an identifier is destroyed. This mechanism allows us to reuse indices and to know whether an identifier is alive or not.

The set of evolved.alive functions can be used to check whether identifiers are alive.

Sometimes (for debugging purposes, for example), it is necessary to extract the index and version from an identifier or to pack them back into an identifier. The evolved.pack and evolved.unpack functions can be used for this purpose.

Here is a short example of how to use identifiers:

First, you need to understand that entities and fragments are just identifiers. The difference between them is purely semantic. Entities are used to represent objects in the world, while fragments are used to represent types of components that can be attached to entities. Components, on the other hand, are any data that is attached to entities through fragments.

Here is a simple example of how to attach a component to an entity:

I know it's not very clear yet, but don't worry, we'll get there. In the next example, I'm going to name the entity and fragment, so it will be easier to understand what's going on here.

We created an entity called player and two fragments called health and stamina. We attached the components 100 and 50 to the entity through these fragments. After that, we can retrieve the components using the evolved.get function.

We'll cover the evolved.set and evolved.get functions in more detail later in the section about modifying operations. For now, let's just say that they are used to set and get components from entities through fragments.

The main thing to understand here is that you can attach any data to any identifier by using other identifiers.

Since fragments are just identifiers, you can use them as entities too! Fragments of fragments are usually called traits. This is very useful, for example, for marking fragments with some metadata.

In this example, we create a trait called serializable and mark the fragments position and velocity as serializable. After that, you can write a function that will serialize entities, and this function will serialize only fragments that are marked as serializable. This is a very powerful feature of the library, and it allows you to create very flexible systems.

Fragments can even be attached to themselves; this is called a singleton. Use this when you want to store some data without having a separate entity. For example, you can use it to store global data, like the game state or the current level.

The next thing we need to understand is that all non-empty entities are stored in chunks. Chunks are just tables that store entities and their components together. Each unique combination of fragments is stored in a separate chunk. This means that if you have two entities with health and stamina fragments, they will be stored in the <health, stamina> chunk. If you have another entity with health, stamina, and mana fragments, it will be stored in the <health, stamina, mana> chunk. This is very useful for performance reasons, as it allows us to store entities with the same fragments together, making it easier to iterate, filter, and process them.

Here is what the chunks will look like after the code above has executed:

Usually, you don't need to operate on chunks directly, but you can use the evolved.chunk function to get the specific chunk.

The evolved.chunk function takes one or more fragments as arguments and returns the chunk for this combination. After that, you can use the chunk's methods to retrieve their entities, fragments, and components.

Full example:

Every time we insert or remove a fragment from an entity, the entity will be migrated to a new chunk. This is done automatically by the library, of course. However, you should be aware of this because it can affect performance, especially if you have many fragments on the entity. This is called a structural change.

You should try to avoid structural changes, especially in performance-critical code. For example, you can spawn entities with all the fragments they will ever need and avoid changing them during the entity's lifetime. Overriding existing components is not a structural change, so you can do it freely.

The evolved.spawn function allows you to spawn an entity with all the necessary fragments. It takes a table of components as an argument, where the keys are fragments and the values are components. By the way, you don't need to create this components table every time; consider using a predefined table for maximum performance.

You can also use the evolved.clone function to clone an existing entity. This is useful for creating entities with the same fragments as an existing entity but with different components.

Another way to avoid structural changes when spawning entities is to use the evolved.builder fluid interface. The evolved.builder function returns a builder object that allows you to spawn entities with a specific set of fragments and components without the necessity of setting them one by one with structural changes for each change.

Builders can be reused, so you can create a builder with a specific set of fragments and components and then use it to spawn multiple entities with the same fragments and components.

The library provides all the necessary functions to access entities and their components. I'm not going to cover all the accessor functions here, because they are pretty straightforward and self-explanatory. You can check the API Reference for all of them. Here are some of the most important ones:

The evolved.alive function checks whether an entity is alive. The evolved.empty function checks whether an entity is empty (has no fragments). The evolved.has function checks whether an entity has a specific fragment. The evolved.get function retrieves the components of an entity for the specified fragments. If the entity doesn't have some of the fragments or if the fragments are marked with the evolved.TAG, the function will return nil for them.

All of these functions can be safely called on non-alive entities and non-alive fragments. Also, they do not cause any structural changes, because they do not modify anything.

The library provides a classic set of functions for modifying entities. These functions are used to insert, override, and remove fragments from entities.

The evolved.set function is used to set a component on an entity. If the entity doesn't have this fragment, it will be inserted, with causing a structural change, of course. If the entity already has the fragment, the component will be overridden. The function should not be called on non-alive entities, because it is not possible to set any component on a destroyed entity, ignoring this can lead to errors. The Debug Mode can be used to check this kind of error.

Use the evolved.remove function to remove fragments from an entity. If the entity doesn't have some of the fragments, they will be ignored. When one or more fragments are removed from an entity, the entity will be migrated to a new chunk, which is a structural change. When you want to remove more than one fragment, pass all of them as arguments. Do not remove fragments one by one, as this will cause a structural change for each fragment. The evolved.remove function will ignore non-alive entities, because post-conditions are satisfied (destroyed entities do not have any fragments, including those that we want to remove).

To remove all fragments from an entity, use the evolved.clear function. This function will remove all fragments at once, causing only one structural change. The evolved.clear function does not destroy the entity, it just removes all fragments from it. The entity after this operation will be empty, but it will still be alive. You can use this function to clear more than one entity at once, passing them as arguments. The function will ignore empty and non-alive entities.

To destroy an entity, use the evolved.destroy function. This function will remove all fragments from the entity and free the identifier of the entity for reuse. The evolved.destroy function will ignore non-alive entities. To destroy more than one entity, pass them as arguments.

The library has a debug mode that can be enabled by the evolved.debug_mode function. When the debug mode is enabled, the library will check for incorrect usages of the API and throw errors when they are detected. This is very useful for debugging and development, but it can slow down performance a bit.

The debug mode is disabled by default, so you need to enable it manually. I strongly recommend doing this in the development environment. You can even leave it enabled in production, but only if you are sure the performance is acceptable for your case.

One of the most important features of any ECS library is the ability to process entities by filters or queries. evolved.lua provides a simple and efficient way to do this.

First, you need to create a query that describes which entities you want to process. You can specify fragments you want to include, and fragments you want to exclude. Queries are just identifiers with a special predefined fragments: evolved.INCLUDES and evolved.EXCLUDES. These fragments expect a list of fragments as their components.

The builder interface can be used to create queries too. It is more convenient to use, because the builder has special methods for including and excluding fragments. Here is a simple example of this:

We don't have to set both evolved.INCLUDES and evolved.EXCLUDES fragments, we can even do it without filters at all, then the query will match all chunks in the world.

After the query is created, we are ready to process our filtered by this query entities. You can do this by using the evolved.execute function. This function takes a query as an argument and returns an iterator that can be used to iterate over all matching with the query chunks.

As you can see, evolved.execute_iterator returns a chunk, a list of entities in the chunk, and the number of entities in this chunk. We already know how to use chunks, so we can use the chunk's methods to retrieve the components of the entities in the chunk, change them, and so on.

Now we know that structural changes are not allowed during iteration, but what if we want to make some structural changes after the iteration is finished? For example, we might want to remove some fragments from entities after we have processed them, or we might want to spawn new entities while processing existing ones. To do all of this, we can use deferred operations.

The evolved.defer function starts a deferred scope. This means that all changes made inside the scope will be queued and applied after leaving the scope. The evolved.commit function closes the last deferred scope and applies all queued changes. These functions can be nested, so you can start a new deferred scope inside an existing one. The evolved.commit function will apply all queued changes only when the last deferred scope is closed.

The library provides a set of functions for batch operations. These functions are used to perform modifying operations on multiple chunks at once. This is very useful for performance reasons.

These functions are similar to the common modifying operations, but they take a query as an argument instead of an entity. Here is a classic example that provides a huge performance boost when applied.

In all other respects, batch operations behave the same way as the common modifying operations that we have already covered. Of course, they can also be used with deferred operations.

Usually, we want to organize our processing of entities into systems that will be executed in a specific order. The library has a way to do this using special evolved.QUERY and evolved.EXECUTE fragments that are used to specify the system's queries and execution callbacks. And yes, systems are just entities with special fragments.

The evolved.process function is used to process systems. It takes systems as arguments and executes them in the order they were passed.

To group systems together, you can use the evolved.GROUP fragment. Systems with a specified group will be processed when you call the evolved.process function with this group. For example, you can group all physics systems together and process them in one evolved.process call.

Systems and groups also can have the evolved.PROLOGUE and evolved.EPILOGUE fragments. These fragments are used to specify callbacks that will be executed before and after the system or group is processed. This is useful for setting up and tearing down systems or groups, or for performing some additional processing before or after the main processing.

The prologue and epilogue fragments do not require an explicit query. They will be executed before and after the system is processed, regardless of the query.

Sometimes you want to have a fragment without a component. For example, you might want to have some marks that will be used to mark entities for processing. Fragments without components are called tags. Such fragments take up less memory, because they do not require any components to be stored. Migration of entities with tags is faster, because the library does not need to migrate components, only the tags themselves. To create a tag, mark the fragment with the evolved.TAG fragment.

The library provides a way to execute callbacks when fragments are set, assigned, inserted, or removed from entities. This is done using special fragments: evolved.ON_SET, evolved.ON_ASSIGN, evolved.ON_INSERT, and evolved.ON_REMOVE. These fragments are used to specify the callbacks that will be executed when the corresponding operation is performed on the fragment.

Use evolved.ON_SET for callbacks on fragment insert or override, evolved.ON_ASSIGN for overrides, and evolved.ON_INSERT/evolved.ON_REMOVE for insertions or removals.

Some fragments should not be cloned when cloning entities. For example, evolved.lua has a special fragment called evolved.PREFAB, which marks entities used as sources for cloning. This fragment should not be present on the cloned entities. To prevent a fragment from being cloned, mark it as unique using the evolved.UNIQUE fragment trait. This ensures the fragment will not be copied when cloning entities.

In some cases, you might want to hide chunks with certain fragments from queries by default. For example, the library has a special fragment called evolved.DISABLED that behaves this way. This fragment is marked with the evolved.EXPLICIT fragment trait, which means it will be hidden from queries unless you explicitly include it. This is useful for fragments that are used for internal or editor purposes and should not be exposed to queries by default.

Additionally, the evolved.PREFAB fragment is also marked with the evolved.EXPLICIT fragment trait. This prevents prefabs from being processed in queries at runtime. Prefabs are used only for cloning entities, so they should not be processed by default.

Often, we want to store components as tables, and by default, these tables will be shared between entities. This means that if you modify the table in one entity, it will be modified in all entities that share this table. Sometimes this is what we want. For example, when we want to share a configuration or some resource between entities. But in other cases, we want each entity to have its own copy of the table. For example, if we want to store the position of an entity as a table, we don't want to share this table with other entities. Yes, we can copy the table manually, but the library provides a little bit of syntactic sugar for this.

To avoid this, evolved.lua provides a fragment trait called evolved.DUPLICATE. This trait expects a function that will be called when the component of this fragment is set. The function should return a new table to be used as the component for the entity. This way, each entity will have its own copy of the table, and modifying one entity will not affect the others.

To make this example clearer, we will also use the evolved.DEFAULT fragment trait. This trait is used to specify a default value for the component. The default value will be used when the component is not set explicitly.

Typically, fragments remain alive for the entire lifetime of the program. However, in some cases, you might want to destroy fragments when they are no longer needed. For example, you can use some runtime entities as fragments for other entities. In this case, you might want to destroy such fragments even while they are still attached to other entities. Since entities cannot have destroyed fragments, a destruction policy must be applied to resolve this. By default, the library will remove the destroyed fragment from all entities that have it.

The default behavior works well in most cases, but you can change it by using the evolved.DESTRUCTION_POLICY fragment. This fragment expects one of the following predefined identifiers:

• evolved.DESTRUCTION_POLICY_DESTROY_ENTITY will destroy any entity that has the destroyed fragment. This is useful for cases like the one above, where you want to destroy all entities when their world is destroyed.

• evolved.DESTRUCTION_POLICY_REMOVE_FRAGMENT will remove the destroyed fragment from all entities that have it. This is the default behavior, so you don't have to set it explicitly, but you can if you want.