Engineering: A Technical Exploration of Argonaut's Notifications System

This article is written with massive help from Prajjwal Dimri who built the Notifications feature.

We are excited to add Notifications to Argonaut. Notifications allow users to get instant updates on any build and deploy actions that occur in a user’s org. This blog is a walk-through of our approach, details of our architecture, along with use cases for this feature.

There are three layers in our architecture: The transformation layer, the processing layer, and the fanout layer. We built the transformation and processing layer in house and used Novu for the fanout layer. It helped us save time by avoiding reengineering this component, which is fairly consistent in most notification systems across various products. It provided us with additional benefits:

1. Easy integrations with communication channels that our users are on.
2. Web widget out of the box, making the integration into Argonaut seamless.
3. User level control for subscribe/unsub for in-app and email notifications.

Notifications play a crucial role in keeping you and your team informed about build and deploy stage updates in real-time. We recognize that the effectiveness of notifications hinges on their promptness, delivery through optimal channels, and appropriate frequency. Consequently, we have designed a customizable notifications system that allows you to receive only the desired notifications through your preferred medium, such as Slack, Teams, Discord, or email.

Use cases

You get a lot of useful information at a glance! The notifications look something like this. And clicking on any notification takes you directly to the respective screen.

We currently provide different kinds of updates for Application Developers. With the addition of events catering to infra coming soon.

Notifications architecture

Here’s a quick overview of our implementation. The process can be broken down into three layers: The transformation layer, the processing layer, and the fanout layer.

Events

Actions by users, such as creating new apps, updating a build or deploy config, and triggering a pipeline, are all considered to be events. We use event driven programming in our backend, where our internal services communicate with each other through publishing and consuming events from a queue. For example, when a build is triggered, running, gets completed or fails, the build service publishes these events to the queue which can be consumed by various other services.

Transformer

The notification transformer listens to a particular topic in the queue, processes the message, and transforms it into a standard format (including information about the org and the user).

Transformers have the capability to deal with various types of events getting generated in the system and will also have the logic to assign priorities to various types of events. (priority queue coming soon)

Standard notification format

We use Novu subscriber in the fanout layer. Therefore, we internally map each org/user with the Novu subscriberId.

Processor

The processor takes the information about the user and the org and retrieves the map, and changes the ID to match the Novu subscriber ID, which is required in the next step to connect with the Fan Out layer and ensure the notifications are routed to the right users.

Fanout layer

The final layer is the Fanout layer, which is managed by Novu. Using Novu made our integration with Slack, Teams, email, and Discord a breeze. The fanout layer (Novu) receives the payload and sends it to the subscribers (members in the same workspace).

Note: Every user and org ID is mapped out to respective subscriberIds.

func (d defaultFanOutService) TriggerFanOut(ctx pkgCommons.IContext, eventId string, subscriberIds []string, payload map[string]interface{}) errors.IError {

novuClient := d.getNovuClient()

_, err := novuClient.EventApi.Trigger(context.Background(), eventId, novu.ITriggerPayloadOptions{

To: subscriberIds,

Payload: payload,

})

Journey of a Notification

• An event occurs and is published in Argonaut’s internal backend service.
• The event is added to the SNS queue.
• The appropriate mapped notification transformer retrieves the event.
• Transformer decides if the event should be processed or not.
• The transformer fetches all the users and organizations for which this event is relevant. For example, In case of a failed build event, the relevant org and its users will be fetched.
• A notification processor (goroutine)is spawned and it picks up the notification.
• Processor loops over the given users and organizations.
• Every iteration, it fetches the subscription preferences, creates a payload to trigger Novu, and dispatches the payload.
• Novu does its magic and sends notifications to users through their selected channels.

Experience using Novu

In our setup, Novu forms the fanout layer, and its main responsibility is sending out notifications to various platforms, e.g., Slack, Email, UI, etc. We are currently using Novu Cloud, which has a free tier supporting up to 10K events per month.

We use Templates offered by Novu, which ties in all the different channels under a single entity. This provides a consistent experience for both the developer and the user, regardless of the channel they choose to receive the notification on.

Entry of notifications and preferences of subscribers is stored in Novu’s MongoDB Database. Notification Aggregation (Digest) is also handled by Novu.

We preferred Novu’s solution over Courier & Knock as it was more economical and it is also an Open Source project.

Novu’s active discord community was supportive when we had queries regarding implementation. We were also able to add our contributions to their Go lang SDK.

We’ve also used Novu’s front-end react client to show notifications within the Argonaut UI. It was easy to use and configure according to our brand colors and design, the configuration of which took under 1 hour.

Upcoming features

• External event integrations where we would add a Webhook processor to receive notifications from external services and third-party tools.
• Adding a priority queue so that high-priority issues like pod failures and cost escalations are shared with the user instantly.
• Notifications for more events catering to different types of users. Such as infra updates, cost, pod utilization, billing, etc.

Conclusion

In summary, adding Notifications to Argonaut has greatly improved the user experience by giving quick updates on build and deploy actions. Using Novu, we've created a smooth and personalized notification system with options for updates through channels like Slack, Teams, Discord, or email.

Our implementation includes the Process layer, Transform layer, and Fan out layer, making sure each notification is useful, prompt, and correct. Working with Novu has led to a more cost-effective solution. We're sure our users will appreciate this new feature, and we're eager to keep enhancing and growing our offerings to meet the various needs of our users