Avital Trifsik for Memphis{dev}

Posted on Apr 5, 2023 • Edited on May 29, 2023 • Originally published at memphis.dev

Memphis is now GA!

#messagequeu #dataprocessing #datastreaming #memphisdev

Memphis is now GA,
and we do not take this title for granted.

Let's start from the beginning.

Struggling with the engineering part of the legacy brokers and queues planted the idea that it needs to be disrupted.
We dagged up more and realized a very rhetorical fact. Still, one that we usually forget – messaging queues and brokers are a means to an end, not the goal itself, and that understanding open a whole new variety of solutions.

The chosen solution was the most challenging one, but we believe, also the right one – a) It has to be open-sourced. b) It can’t be just an intelligent message broker on steroids. c) It has to offer what we call the “Day 2” operations on top to help build queue-based applications in minutes. From the more common ones, which are async communication between microservices, task scheduling, to event-driven applications, event sourcing, data ingestion, system integration, log collecting, and forming a data lake
With that understanding in mind, we formed the vision of Memphis – an intelligent and frictionless message broker that enables an ultra-fast development of queue-based applications for developers and data engineers.

From vision to GA.

Memphis beta version released on May 15th, 2022.
We focused on the foundations of the ecosystem, integrating with NATS internals, designing memphis to run natively on Kubernetes and cloud-native environments, out-of-the-box everything - from monitoring, dead-latter station, schema validation, real-time observability, and more.
With each release, the bug cycles became shorter and smaller, and we, as a team and product, became more intelligent by carefully understanding and listening to our users. By doing that, Memphis reached a solid and stable GA, and not less importantly, suitable for most developers and not just those who share our original challenges.
April 2nd, the GA release of the 1st part of Memphis.
Memphis GA stands for a solid, stable, and secure foundation for the future to come with zero known bugs and ready for production.

Some insights from the last eight months.

Avg time from installation to data ingestion takes 5 minutes.
We grew from 0 to over 5000 deployments.
50 new contributors.
Users report production usage before the GA release.
Use cases from async communication between microservices, event-driven applications, event sourcing, data ingestion, system.
integration, log collecting, security events, to forming a data lake
Schemaverse has been a game changer to many of our users.
The most used SDK is Go.
Cost and simplicity have been major factors in replacing existing tech with Memphis.

The future to come.

I mentioned 1st part, so there is a 2nd part.
Memphis’ 1st part is the storage layer, the message broker with all its benefits as we know it today, and will continue to evolve dramatically over the coming releases. We will also push hard on GitOps, automation enablement, and reconstructing some of the APIs so they can be modular and open for the community to self-implement new ones. Last but not least – multi-tenancy, partitions, read-replicas, and more.

Memphis’ 2nd part is all about helping developers and data engineers build valuable use cases and queue-based applications on top of Memphis. More on that in the coming weeks.