Show HN: Nixiesearch, an open-source alternative to Elasticsearch Serverless

Nixiesearch is a modern search engine that runs on S3-compatible storage. We built it after dealing with the headaches of running large Elastic/OpenSearch clusters (here's the blog post full of pain), and here’s why it’s awesome:

• Powered by Apache Lucene: You get support for 39 languages, facets, advanced filters, autocomplete suggestions, and the familiar sorting features you’re used to.
• Decoupled S3-based storage and compute: There's nothing to break. You get risk-free backups, upgrades, schema changes and auto-scaling, all on a stateless index stored in S3.
• Pull indexing: Supports both offline and online incremental indexing using an Apache Spark based ETL process. No more POSTing JSON blobs to prod cluster (and overloading it).
• No state inside the cluster: All changes (settings, indexes, etc.) are just config updates, which makes blue-green deployments of index changes a breeze.
• AI batteries included: Embedding and LLM inference, first class RAG API support.

Search is never easy, but Nixiesearch has your back. It takes care of the toughest parts—like reindexing, capacity planning, and maintenance—so you can save time (and your sanity).

!!! note Want to learn more? Go straight to the quickstart and check out the live demo.

• Nixiesearch is not a database, and was never meant to be. Nixiesearch is a search index for consumer-facing apps to find top-N most relevant documents for a query. For analytical cases consider using good old SQL with Clickhouse or Snowflake.
• Not a tool to search for logs. Log search is about throughput, and Nixiesearch is about relevance. If you plan to use Nixiesearch as a log storage system, please don't: consider ELK or Quickwit as better alternatives.

Our elasticsearch cluster has been a pain in the ass since day one with the main fix always "just double the size of the server" to the point where our ES cluster ended up costing more than our entire AWS bill pre-ES [HN source]

When your search cluster is red again when you accidentally send a wrong JSON to a wrong REST endpoint, you can just write your own S3-based search engine like big guys do:

• Uber: Lucene: Uber’s Search Platform Version Upgrade.
• Amazon: E-Commerce search at scale on Apache Lucene.
• Doordash: Introducing DoorDash’s in-house search engine.

Nixiesearch was inspired by these search engines, but is open-source. Decoupling search and storage makes ops simpler. Making your search configuration immutable makes it even more simple.

How it's different from popular search engines?

• vs Elastic: Embedding inference, hybrid search and reranking are free and open-source. For ES these are part of the proprietary cloud.
• vs OpenSearch: While OpenSearch can use S3-based segment replication, Nixiesearch can also offload cluster state to S3.
• vs Qdrant and Weaviate: Not a sidecar search engine to handle just vector search. Autocomplete, facets, RAG and embedding inference out of the box.

Get the sample MSRD: Movie Search Ranking Dataset dataset:

Create an index mapping for movies index in a file config.yml:

1. We use ONNX Runtime for local embedding inference. But you can also use any API-based SaaS embedding provider.
2. Any SBERT-compatible embedding model can be used, and you can convert your own

Run the Nixiesearch docker container:

Build an index for a hybrid search:

Send the search query:

You can also open http://localhost:8080/_ui in your web browser for a basic web UI:

For more details, see a complete Quickstart guide.

This project is released under the Apache 2.0 license, as specified in the License file.