The Ops Community ⚙️: Lob Developers

To Infinity & Beyond: Our Migration to Nomad is Complete

Lob Developers — Fri, 30 Dec 2022 21:41:49 +0000

Lob’s core API has been fully migrated to HashiCorp's Nomad, Lob’s Next Generation service platform. This is a major milestone for the Nomad Project, the Platform Team, and Lob Engineering. This migration is the culmination of a year of R&D, months of practice migrating other Lob services, and weeks of work on this particular service. It’s absolutely worth celebrating for the complexity and customer impact.

TLDR: Lob’s core API has migrated from Convox to Nomad. Our API is running better than it was before and with Nomad we have the tools to tackle platform issues that were previously not possible.

Why?

The way we run code for our API and dozens of supporting tasks has organically grown based on the needs and constraints of the business. Most apps at Lob ran on Convox, a straightforward service management tool we license, but a handful of workloads have needs that are not met by Convox! Historically, these run on AWS Lambda for security or scaling reasons, or Heroku for development ease, and some run on ECS for various customization needs.

The point is that this artisanal, organic, Non-GMO architecture was and continues to be a huge burden on Lob Engineering. The Platform team needs to know half a dozen tools very well and the rest of engineering must settle for a narrow feature set or spend weeks ramping up on a different technology. This drags on all new products, features, and bug fixes resulting in lower engineering velocity and a lot of hair-pulling.

It was clear back in 2021 that Lob needed to consolidate how we run code and none of our current tools were up to the task; it wasn’t a matter of if Lob would upgrade to something new, but when. The Platform team kicked off a research project to find Lob’s next service platform. Forever ago (back in 2019) we investigated migrating to Kubernetes, a popular but notoriously difficult-to-manage tool for this sort of thing, but that project fizzled out for many reasons, forcing us to consider something else. We chose Nomad which offers a comparable feature set to Kubernetes in a much more streamlined package. Nomad is developed by HashiCorp, a leader in the DevOps space, and is used by companies like Pandora, Cloudflare, Internet Archive, and Roblox.

Nomad is able to meet all of Lob’s business needs today and into the future, allowing us to consolidate where we run most of our code onto one platform. We are able to specialize in this tool, create custom workflows to meet Lob’s changing needs, and tune the underlying architecture to outperform our old solution(s). Where before we had to stretch ourselves thin, or live without some features, Nomad is a one-stop shop.

What does this get us today?

Our core API running on Nomad has performance parity with our old container orchestrator, but it’s even better in some ways. Here’s how:

Lob releases happen via GitHub Actions, giving us better visibility into when the last release was, what got deployed, if it failed, and who kicked it off.
Ad-hoc tasks like Database Migrations and Scripts are also managed in GitHub Actions, moving even more processes off of your laptop and onto the cloud.
Nomad offers a much richer UI, enabling Lobsters to gain better visibility into how their app is working or debug their app when it isn’t.
Nomad’s autoscaling is much more customizable, allowing us to scale our API more responsively to meet spikes in customer requests.
Tag-driven releases allow us to audit when code was deployed and promoted to customers.

What does this get us tomorrow?

The real reason to migrate to Nomad is for the laundry list of shiny new features that unlock major potential down the road…

Hosting containers on AWS Spot Instances should save us money on our AWS bill.
Middleware for handling authentication and other business needs at the edge.
Canary deployments, finding regressions before rolling them out to all customers.
Federated services, hosting our apps in AWS regions closer to our customers.
Autoscaling based on custom metrics like workload prediction.

The final piece of the puzzle

Most of our API’s migration to Nomad happened in August, but the key feature of autoscaling was not working as expected. This turned out to be a bug in Nomad which James Douglas tracked down. The issue was recently fixed and autoscaling works as expected, completing the migration!

This migration, as with all of Lob’s Nomad migrations, is intended to be zero downtime. This means we would slowly migrate traffic to Nomad and roll back if we noticed any regressions in order to minimize customer impact; in most cases, customers cannot tell that this type of change took place. Autoscaling was an optional feature—we could just run Lob at max—but that would be a waste of resources and wouldn't really bring us to parity with the old Convox deployment.

50+ services have now been migrated successfully!

Countless Kudos

This is the type of project that cannot happen in a vacuum. The Platform team is incredibly thankful for all of the Lobsters past and present who have helped us out. You have lent us your time, expertise, patience, and trust in a project that really needed it. While it’s easy to move on to the next big project, moving our core API to Nomad is an accomplishment worth taking a moment to celebrate.

Special thanks to Senior Platform Engineer Elijah Voigt.

I Know What You Shipped Last Summer

Lob Developers — Tue, 20 Dec 2022 17:59:25 +0000

A hackathon is the perfect embodiment of several of Lob’s core values including Be bold, Move fast, take action, and Level up! It’s an opportunity to step outside the daily hustle, collaborate, innovate, and problem-solve. The only thing more fun than a hackathon is a themed hackathon—especially when the theme is Halloween!

We want to acknowledge the hard work our Product and Engineering teams put in by highlighting their projects; the following is a technical recap of Lob’s Halloween Hackathon 2023.

SQS-consumer gets a glow up

Many of our projects focused on improving the developer experience at Lob, but a few innovative Lobsters created a new open-source library that also benefits our community as well.

AWS SQS, or Simple Queue Service, is a piece of infrastructure offered by Amazon; it’s a powerful tool processing data in a controlled manner. “Sqs-consumer” is an existing open-source library for working with SQS.

Though it’s a very popular library—we use it in Lob API and other services—it is not being maintained (major issues and suggested features are not getting attention) and perhaps most importantly, it’s not been upgraded to the new version of AWS SDK (V3). This prohibits us from upgrading many of our workers, and Lob API as a whole, to V3 of the SDK.

Solution: A new repo for working with SQS @lob/sqs-consumer! It's an open fork of the previous package, but it makes a few key changes:

Upgrade underlying AWS SDK to V3
Upgraded all dependencies
Converted test suite from an outdated version of Mocha to Jest
Fixed challenges with FIFO queues

Even though we now offer a low-code solution to automate direct mail (via our Campaigns product), we will always be a developer-first company. This library is a big win for our engineers, and we look forward to feedback and contribution from the developer community.

Software Engineer, Sishaar Rao (with help from Staff Software Engineer, Landon Barnickle, and Senior Software Engineer, Hanqing Chen) tied for second place for this ingenious project.

o11y is the bee’s knees

Another hackathon project focused on increasing our understanding of the services at play at Lob. Staff Software Engineer Rich Sevoria and Senior Software Engineer Benny Kitchell teamed up to tackle observability.

Lob’s rendering engine transforms HTML templates into print-ready PDFs, at scale. Every so often, we’ll encounter a glitch, and debugging can be a very manual—and painful—process. Most of the logging is contained within an internal service so any one individual lacks full visibility into all the services that are connected to each other and where they are failing. Ideally, you could visualize what's taking place, how long it takes, and in what order.

Sevoria proposed a two-part solution.

First, utilizing Open Telemetry, or OTel, ”a vendor-neutral open-source Observability framework for instrumenting, generating, collecting, and exporting telemetry data such as traces, metrics, logs.” Using traces and spans, we can get detail and visualization of all the events that took place across multiple services that aren't connected in any way outside of that they're all our services.
Then we can use Honeycomb for a deep-dive analysis of this information.

Unlike Datadog (which we use at Lob quite a bit), you don't need to know what axis you want to analyze in advance. (Anybody who spends enough time with Datadog knows that if you wanted to, for example, search aggregate information by a given data set, you better have known that already and have created a facet for it; if you haven't, you're not going to be able to get an accurate read off of it.) You can aggregate on any dimension at any time you want, which is fantastic for analysis.

Sevoria noted two other benefits of exploring Honeycomb for data analysis:

using industry-standard typically results in less technical debt (in form of a really tightly coupled adherence to a specific vendor's implementation), and
Honeycomb has a very transparent and easy-to-understand pricing model.

Simplifying local development

In addition to hops along the way to delivery, during creation, each mailpiece takes a trip through a network of distributed services. This means Lob services are difficult to stitch together when developing locally. We've historically called staging instances from our local environments to simulate the production behavior. Unfortunately, this process breaks down when the interaction is asynchronous or when multiple services need to be updated simultaneously.

This problem was introduced with ominous music and masterful editing. Remember the scene from “Saw” when Jigsaw says there is only one key? Masterful editing had Jigsaw following up with, “to find the key, you must integrate services in Docker.”

The team identified three reasons why it’s so difficult to integrate in Docker: differences in naming conventions, port bindings, and overlapping resources. Lob uses AWS so we communicate with each other through queues. This can get quite complex, (previously queue messages are problematic for example), but the team uncovered a way to simplify it.

They proposed standardizing starting conventions, staggering our ports, and sticking with one version (or one running container) for joint resources like one AWS stack and Redis. Before vs. After:

Each service can be started separately via a new startup script. This ensures shared localstack resources are started, adds the queues/buckets, etc., to the shared AWS localstack that is needed, and starts up only needed services in the Docker network: shared. Port binding conflicts can be avoided by referring to the new (very high-tech) shared Google spreadsheet.

This has the potential to standardize how Lob’s apps communicate with each other. This aligns with similar initiatives within Lob architecture and a continued push towards operating as a unit or as a whole ecosystem, not just individual teams.

Major kudos to Staff Software Engineers Krys Flores, Ken Pflum, and Adrian Fallerio, Senior Software Engineer David Nutting, and Software Engineer Nick Perri—their efforts resulted in a tie for second place.

GitHub productivity hack

Easily winning the best costume award was Senior Software Engineer, David Nutting, for his “Lob” Zombie getup.

Nutting kicked off our presentations with a GitHub productivity hack. He outlined the frustrating and time-consuming scenario where you are waiting on your code to be approved, then manually executing a merge, only to be served with the “This branch is out-of-date with the base branch” message (sometimes not just once, but twice).

As it turns out, automatically merging a pull request is a feature that's built right into GitHub. To turn it on, you just have to open up the repo settings and check two boxes. That's it!

Though not a feature of GitHub proper, there are also Github Actions that will essentially press that “Update branch” button for you. So when any other merge happens to main, a "push" event will trigger some code to tickle your branch and pull new changes. (If you are concerned about future conflicts, you can set up as many notifications as you want in GitHub.)

While better for some use cases than others, this certainly is a time-saver.

Changing of the guard

Karmbir (KC) Chima was on-call for a portion of the hackathon; this ended up being the inspiration for his project. He too was interested in saving time. At the end of each on-call shift, the primary engineer—or “goalie”—is responsible for a comprehensive handoff. This involves documenting all issues (resolved and ongoing) and other detail; essentially duplicating all the info captured in the tickets. But let’s be honest, at the end of two weeks, not everyone has the time/energy/memory to do this at a high level. Realizing that Notion can natively sync with Jira, the solution was obvious: Add an “on-call” label to the request form (a Slack Workflow that creates a Jira ticket); this automatically syncs into the Notion document, eliminating the need for recall and duplicate data entry.

Run, Dora, run

Staff Engineer Guy Argo also had efficiency on the brain. Lob’s address verification software, affectionately named “Dora,” is CASS-certified, which is a huge benefit to our customers, but does require quite a bit of work to maintain. For the latest round of certification, we have about 150,000 tests to run.

To get things running a bit quicker, he persisted all the data for the tests in the database but was still looking to shortcut the endless send/retrieve in Elasticsearch. So why not Redis? 20 lines of code and 1 cached index later, he saw a 20% improvement in runtime on the large test dataset—a perfect productivity hack! Correr, Dora, Correr!

Show me what you’re working with

The Partner Management & Tools engineering team has the unofficial mission to make life easier for Lob’s partners and our internal Partner Operations group; both have expressed the need for greater inventory visibility. Together the team built two new UIs that allow visibility of inventory by partner, and inventory change by day. Previously only available in Datadog; this information is now displayed in an easy-to-digest way; areas of concern can be quickly identified, as well as trends.

This was a huge win for both our partners and Lob, and for their efforts, the team took home the gold ribbon. Congratulations to Engineering Managers Adam Stodgill and Andrew Jorczak, Senior Software Engineers Martin Han, Tim Bright, and Zac Leids, and Software Engineers James Cho and Jessica Ho.

Filter what you’re working with

Up next, to make a case for “there's no small PRs, only small teams” was solo artist Staff Software Engineer, Landon Barnickle. Our Partner Ops team has also expressed a pain point around filtering. To make their lives easier, Landon added an admin route for managing filtered partners using a wrapper for our Redis cache, or as he put it: List, Add, Remove, WIN! He reminded our engineers of the importance of working closely with the end users; often little changes can make a big impact.

See the future, I can

Our Routing and Delivery engineering team (more commonly known by the awesome acronym RAD) also supports our partners. For example, we currently have a tool for our team to create dynamic rulesets around routing to our print partners. To use a very simple example, let’s say we wanted to route all mailpieces from a customer to a specific print partner. Using GitHub Actions, our team can create this rule quickly and easily—but they do so in production.

The downside of such agility is we don't have a good way of predicting the effect of any given change. Should the print partner in the above example be at capacity, or if there is a conflict with another ruleset (more common), we must make another adjustment.

Ken Pflum, Staff Engineer, developed a way to allow us to A/B test; in short, we can now work off a particular branch to test rule sets, then merge that branch into main. This predictive ability reduces risk and still allows us to adapt quickly.

A picture is worth a thousand words

Visibility was a common theme in this year’s hackathon. Lob sends millions upon millions of mailpieces each year which results in a lot of data. For those within the company, there's no easy way to digest that data. Looker dashboards have limitations and even those with SQL chops may run into difficulties writing queries.

Engineers on the RAD team aimed to provide a couple of visualizations depicting mail volume and our routing approach. Why visualizations? They are easier to digest and more descriptive than code and they help surface patterns and anomalies.

Sample data was used to showcase mail density over the last month across the US:

Using data from a sample postcard campaign, they also presented a simulation of what a routing profile could look like on the US map. It follows the campaign getting routed through Lob: The bigger bubbles represent Lob’s print partner facilities and the smaller bubbles represent the USPS facilities; the links showcase the USPS facilities where our partners sent our mail.

Props to Senior Software Engineer Sachin Muralidhara and Software Engineer Joseph Villanueva, with nods to Staff Software Engineer Landon Branickle and Engineering Manager Adam Stodgill. The team at large is excited to build on this work.

The coolest traffic cop

In another effort to standardize development and operations, Lob has just wrapped up our container orchestration migration from Convox to HashiCorp’s Nomad, led by Senior Platform Engineer Elijah Voigt. In this new ecosystem, one feature available to us is Consul Service Mesh (a feature of Consul, which is part of our Lob Nomad stack).

With Consul Service Mesh, we can have each of our services declare what their dependencies are, and enforce only traffic from those services. So you can tell your container orchestrator: this service should only accept traffic from this other service, and anything else should be blocked (and that service is encrypted). As we scale up, this offers us the same zero trust security much larger enterprises typically employ.

In addition to setting us up for a better security posture, Service Mesh also provides a greater understanding of how our services work together. Topologies allow you to quickly identify dependencies and interactions, and details are available for a deep dive.

A journey of a thousand miles starts with a single step

Senior Director, of Strategic Business Development, Tyler Dorenburg, and Senior Product Manager, Andrew Reagan are passionate about bringing new functionality to our customers. Working closely with our engineering team they continue to drive innovation.

Informed Delivery by USPS is a free daily digest email with digital scans of each mailpiece you will receive each day. Additionally, mailers can add a clickable advertisement underneath each mailpiece scan.

As of September 1st, USPS adjusted the requirements making it possible for Lob to support this functionality for our customers and their mail campaigns in the near future! The technical details of how we plan to implement this are both mindboggling and proprietary so we won’t share that here, but Staff Software Engineer, Landon Barnickle, and Software Engineer, Gage Guzman, used the hackathon as an opportunity to dive into some of the prework (like digesting a 92-page PDF with XML specs—sounds terrifying!).

Go big or go home

Arguably the most frightening moment of the hackathon demos was when Senior Infrastructure Engineer, James Douglas, showed the group our AWS bill. Ek! We are storing over 5 petabytes of data in the form of billions of objects. Our tables have grown so big that we're timing out on a lot of queries.

We developed a service called “Expunge” which deletes expired data on an ongoing basis but has a hard time working through our current backlog. It’s like using a lawn mower when a feller buncher is what’s needed. (You are not alone if you had to look up “petabyte” and/or “feller buncher.”)

True to the Lob value Be Bold, James went in search of the tsar bomba, that is, a one-time big-bang approach to clear all the old data.

He initially explored Amazon S3 Lifecycle which is a built-in wipe but came across a number of limitations.

For example, why not just delete everything that's X days old, right? Wrong. There are special circumstances where we want/need to keep data for longer.
What if we tagged them, then have S3 delete the rest? Wrong again; turns out you cannot exclude a tag in the filters.
Move them to a different storage class? Nope. Can't target only a storage class.
How about we evacuate the files, then copy them back in so it updates the date (to fall within the “keep” period)? Nope. Innocent bystanders would be caught in the slaughter, like templates, which are timeless.

Alas, the tsar bomba approach would be too indiscriminate. Since the goal is to target each S3 individually, James landed on the following instead:

He attacked the database first by carefully moving everything that qualifies for deletion into a new table and indexing it; this took about 19 hours. From there, you can use Expunge’s core functionality to attack this new table to delete resources. Testing proved this process would take mere minutes; minutes that will save us major moola. This is a huge win for Lob.

With S3, an ounce of prevention is worth a pound of cure" —Douglas.

Stripe Upgrade

Proactive upgrades are also worth their weight in gold. Software Engineers Sage Farrenholz and Patrick Thulen on our Billing team tackled an upgrade to Stripe, the system we use to automate and process payments. True to theme, Sage noted our legacy Stripe integration was pretty “scary”: we were using an older version, and we wanted to move from Charges API to the newer Payment Intents.

The work was fairly straightforward: They bumped Stripe (Node) Client package to 10.15.0, bumped Stripe API to the latest version 2022-10-0, and moved over to Payment Intents, but this involved multiple breaking changes and an awful lot of testing.

While not the sexiest of projects (although an accidental dropping of the “e” from “Stripe” on one of the presentation slides would suggest otherwise), this brings tremendous value to our Lob and our customers.

Wrap Up

After 48+ hours of non-stop hacking, our Lobsters looked a little more like ghouls. That said, each and every participant, and all those who got to witness the final presentations, had smiles on their faces when all was said and done. Hopefully, they were sincere…

“No matter the situation, always wear a smile.” –The Joker

Want more?

We love a good theme; read more about our spring hackathon, inspired by a favorite 80's movie: Hack to the Future.

Mail your holiday cards programmatically

Lob Developers — Tue, 06 Dec 2022 20:57:04 +0000

It’s Holiday Card season! But my hand hurts just thinking about addressing all those envelopes, and I can’t remember the last time I bought stamps…let’s be real, ain’t nobody got time for that.

Fortunately, I happen to work for a company that sends direct mail programmatically. Many developers integrate with Lob using one of our SDKs in their preferred programming language, but we recently launched our low-code Campaigns and I thought I’d give it a whirl.

The following is a step-by-step tutorial so you too can create your own full-color 6”x9” holiday postcard (to be printed and mailed by Lob) for under a dollar each.

Here is a sample of the postcard we will create.

Select and prep your photos

Choose two of your favorite photos: one landscape (for the front), one portrait (for the back). Use your photo editing tool of choice to format each correctly; I used Preview on my Mac. To ensure that they scale proportionally and print at the highest quality they should be 5x7 inches (2100 pixels x 1500 pixels in landscape or 1500 pixels x 2100 pixels in portrait) and at a resolution of 300 pixels per inch. (If you need more support on this part, and/or are snagging photos from your iPhone, see the Appendix.)

You will then need to host your images somewhere; I used a free account at imgbb.com.

Prep your data

You will then need to bring up the HTML for the front and back of the postcard. Click each link and view the HTML; we’ll use it in a later step).
Next, you will prep your address data. Make a copy of this spreadsheet and input your data:
Do not modify or add any column headers!
Add the full name in the name column
Add the address data (make sure that leading zeroes aren’t being dropped and that the state code is used)
Replace the “Lorem ipsum” in the message column with your own personalized message. Make sure that the message fits within the space by being shorter than the placeholder examples (about 65 characters) You can test your holiday card messaging HERE
Add the links to your front and back images to every row in the image_front and image_back columns (should look something like this: https://i.ibb.co/fpJ84Js/front.png)
Download and save your address file as a .csv.

Create your HTML templates

Now you are ready to create your postcard templates in the Lob dashboard.

First you will need to sign up for a free Lob account.
Once you have registered you will be taken to the dashboard. In the Developer tier there is no cost to test out the Print & Mail APIs, but if you want to send live requests—and have a postcard printed and mailed—it will run you $0.896 a pop.
Payment information can be entered here: Your name (top right) -> Settings -> Payment
Create & save HTML templates for the Front and Back of your postcard
On the left menu bar, Navigate to HTML Templates.
Make sure Live is highlighted!
Click on ‘Create’
Name this ‘Front Template’ and paste in the HTML code for the front. Click ‘Create’ to save.
On left menu, click HTML Templates to go back to main template screen. (You should see your new Front template there under Live templates.)
Once again, ensuring Live is highlighted, click ‘Create’ again. Name this ‘Back Template’ and paste in the HTML code for the back. Click 'Create' to save.
Click HTML templates to go back to main template screen; you should see your new Back template there under Live templates.

** Make sure you see both templates are listed when Live is highlighted before proceeding.

Create and send your postcards

On the left menu bar, navigate to Campaigns; click on “Create Campaign”

Step 1: Configure Campaign

Enter a Name and Description
Campaign Type: Marketing
Mail Type: Postcard; Size: 6x9
Postage references: Return address is not required; Select postage: Up to you, note the listed expected delivery times
Cancellation Window: This is set at 5 minutes; you will not have the option to edit on a Developer plan
Campaign-Level Metadata: Optional

Step 2: Add Audience

Upload your CSV file (created during prep stages), or drag and drop it into the dashboard
Map required address variable: These are mapped automatically.

Step 3: Choose Creative

Under ‘Postcard Front’, select the HTML Template you created from the dropdown menu. (Note a preview will not show your images or messages as they are not mapped yet)
Under ‘Postcard Back’, select the HTML Template you created from the dropdown menu. (Note a preview will not show your images or messages as they are not mapped yet)
Connect your creative to your audience:
Merge variable strictness: Ignore;
Connect/map your merge Variables from HTML to the appropriate column in the CSV: {img_front} to img_front, { img_back} to img_back, {message} to message

Step 4: Review Campaign

If something is not correct, click ‘Previous Step’ to edit. The free Developer tier does not allow you to edit cancellation windows. That said, once you place your order, after 5 minutes, the live request will be sent to Lob. Your postcards will be created and mailed, and you will be charged.

If all looks correct, click ‘Place Order’ and your holiday cards will be on their way! You can view your cards under Campaigns in the Lob dashboard.

Wrap up

Lob was started by two developers who needed to solve a problem, and direct mail APIs were the answer; the goal was to make sending mail just as easy as sending an email. This is a fun personal use case, but Lob is meant to automate direct mail—postcards, letters, checks, self-mailers, and more—at scale. Just like email, each mailpiece can be highly personalized, and for advanced tiers, Mail Analytics is available for each mailpiece to track each postcard along its journey to delivery (and if you included a QR code, you could even track open rate). Check out Lob.com if you want to learn more.

Appendix: How to prep a photo

Email/Download a photo from your iPhone onto your computer. The following steps are shown in the Preview app (Mac) but are a good guideline for any editing program.
Select your back photo (portrait orientation). Under Tools, select Adjust Size.

Change Resolution to 300 pixels/inch.
For this photo (portrait orientation): Change Width to 5 inches; and Height will automatically change. Alternatively, update the Height to 7 inches, and the Width will change. It’s highly likely you won’t end up with a perfect 5x7; that’s ok: Change the field that results in one of the measurements being slightly too big; you will crop in the next step).

Under Edit, Select All. Drag the dotted line to crop the image to the necessary 1500x2100 (as you move the line, the measurements will display; exaggerated below).

Under Tools, Assign Profile select Generic RGB. (Photos must be saved as RGB.)
Make sure your photo is named clearly as the BACK. Save!
Repeat these steps for **the FRONT photo, but in landscape orientation (7” wide x 5” high, 2100 pixels x 1500 pixels).

Starting a new job as a Staff+ Software Engineer? I got you.

Lob Developers — Thu, 01 Dec 2022 17:32:52 +0000

My experience getting hired at Lob at the Staff level made me realize I may have some learnings to share. I've been at the Staff level previously, but that was at a company where I had been promoted up to those levels; I was already very familiar with my team, my company, our business, and our platform, so it was just a natural progression of responsibility for me. But getting my current job was the first time I'd started at a new company at such a level of seniority. And while at least for me, every new job is a major invitation for an Imposter Syndrome attack, this had the potential to be an even greater one. But I fought it, and I was really intentional about how I approached getting up to speed and be effective more quickly and smoothly than I have in previous new job experiences.

So you've landed the job, now what?

You worked hard to get through that interview cycle, you got the job offer, you accepted it, —you won! But then the insidious thoughts can start. "But what if they got it wrong and I'm actually not good enough? What if people there are smarter than me? What if they don't like me? What if their expectations of Staff are different than where I was before and I don't meet them?" The stories can go on and on if you let them.

This is where I stop the stories and I tell myself, "Hi, Imposter Syndrome. I see you. But I'm not going to let you control my life. Even though I'm feeling really nervous right now, here are some facts I can tell myself:”

I got the job. They liked me enough, they saw enough of something in me to offer me the job.
I have a proven track record for performing, getting positive feedback and reviews, and getting promoted. All of those people saw something in me and they can't all be wrong.
All I can ever do is my best. I can work hard, I can try my best, and if that's not good enough, then that's outside of my control.
Worrying about what might be before I have any information to act on isn't going to change or help anything.
There will be a number of people that are smarter than me or have more experience than I do in a number of things, and that's okay. If I'm the smartest person someplace, then who will I learn from? How will I grow?

No one is a know-it-all

Let’s address that last point. Starting at a new company, at a level of some seniority, it can feel like you should be expected to be the expert in all the things. Well, there's this concept that I constantly remind myself of called being T-shaped; inspired by this article in Forbes: Why T-shaped teams are the future of work.

While generalists know a little about a lot of subjects, and I-shaped employees are experts in a single area, a T-shaped person is a subject-matter expert in at least one area and knowledgeable or skilled in several others.

If we lined up several t-shaped engineers in an organization we would have plenty of overlap in general knowledge but bring a different mix of experience and specialization to the table. So, I don’t have to be the expert in everything; we can share the load.

One of the most powerful blog posts I've read, and it really helped me personally with this concept was written by a well-known engineer I respect, Dan Abramov, titled The Things I Don't Know. He very openly and publicly shared with thousands of followers that while he may be an expert in a number of things, there are plenty of things that he doesn't know very well or at all. What was really empowering about that list is there were some things that he didn't know that I did. So if Dan Abramov can be brilliant at some things and struggle with others, I can too.

Behaviors to Avoid

You’re really going to want to show everyone as soon as possible that you know what you’re doing and deserve the title you’ve been given. But, here are some things to watch out for so you don’t end up starting off on the wrong foot and acting like a jerk.

1. Don’t assert things just to show off what you know

It's fine and absolutely appropriate for you to be sharing knowledge and helping to educate others, but just make sure the context is right. **If you find yourself constantly feeling the urge to talk about the things you've done before or point out something you know that doesn't actually add value to the conversation, try to hold back. **This can make you either sound arrogant or desperate if you do it too much. You will have plenty of opportunities to demonstrate your skills and knowledge.

2. Don’t try to change too much too soon—take some time to observe and learn first

Maybe something isn't exactly the way you would've done it or have done it in the past, but that doesn't necessarily mean it's wrong. Make note of these things that seem to go against what you would've done in the past, but resist the urge to jump in immediately and call it wrong. Part of your role is to improve things, and teach others how to do things in a better way, but until you've had time to really get your surroundings, you may not have enough context yet.

Take some time to get the full picture. Gather data, ask questions, but withhold judgment until you've had a little more time to get to know people, processes, and history. Ask questions humbly and from a place of curiosity.

Along that note, pick your battles. When you have that instinct to assert that something should be done differently, continue to ask yourself, “How important is this right now?” Make a note, and you can revisit it later with more context.

3. Resist perfectionism

During this time of high insecurity, you're going to want to project nothing but perfection. But perfectionism is a myth and thinking that you have to be perfect is toxic, not just for you but for the others around you. Part of being a leader is mentoring others, and a portion of that is done by what behavior we are modeling to them. If we show others that we never make mistakes, there's nothing we don't know, we work however many hours it takes to get the job done, we're always available, we're never stressed or tired—I could go on and on—we send that message to others that in order for them to be successful, they have to be that way, too.

It encourages them to set unhealthy, unfair, and unrealistic expectations of themselves. So as much as it goes against that urge to prove yourself to everyone right now, you are doing a greater service to others if you can model some vulnerability. More junior folks need to see that everyone makes mistakes, no matter how much experience you have.

What_ is_ important is demonstrating how you handle yourself when you make a mistake. Calmly own your mistake openly and early. "Here's what happened and here's my game plan for what I'm going to do about it." Take responsibility without being embarrassed or overly apologetic. Just be transparent about it and then get it fixed.

We should foster an environment of continual learning. There were many times when I'd have to say, I don't know how to do that yet, but I'll get back to you. Then I'd use that as an opportunity to go figure that thing out. Don't act like you're just so smart that nothing is hard for you, but rather show others it doesn't have to be scary to do something you don't know how to do.

As software engineers, we need to be comfortable with being uncomfortable and able to figure things out. I see a lot more junior folks trying to stay in the areas they're already familiar with and avoid branching out. They're hanging out in the shallow end of the pool. Show them by example that swimming into the deep end doesn’t have to be intimidating.

Behaviors to Adopt

Now let’s address what you should do to get up to speed quickly and start providing value ASAP.

1. Start taking notes early

Your role as a Staff + Software Engineer is to be a sharpened tool. You bring with you a lot of experience and skills and you can be deployed to accomplish tasks for your team, your manager, or even the engineering organization level. You may be given the responsibility and freedom to work on special projects or tasks that cross team boundaries. You may be given a level of autonomy to decide or at least propose what you think you should be working on.

That said, you can only do so many things at a time. So you have to be very thoughtful about how you prioritize your time and what you choose to spend that time on (more on that in a moment). But as you're learning your way around, start taking notes. You should be keeping your eyes out for opportunities to improve, optimize, and make systems, processes, and people better, but early on, just make notes on all of those things you notice for reference later.

2. Prioritize what you're spending that time on

As you're finding yourself getting caught up on the learning curve, and you're starting to feel more comfortable with your surroundings, that's a good time to start reviewing that list of items you've been building. Is everything on it still something worth spending time on? Can/should any of those things be delegated to others? Whatever's left will need prioritizing.

In the book _Staff Engineer _by Will Larson (which I highly recommend) he references a quadrant of impact versus effort.

“Work on What Matters” is worth a read to understand where you might go astray, but, in short, the goal is to aim for the top half of the quadrant at the “high impact” tasks. Keep checking in with yourself, both before you prioritize your work, and after you’ve started a task, to make sure it’s the right thing at that time for you to be doing.

3. Start meeting people

If you're an introvert or you have some social anxiety this could be really difficult. But I've learned how to push through that anxiety when it's important. Inside, I may be wanting to curl up in a ball and hide, but outside I try to project comfort and confidence; I think that's a crucial skill for being successful as a software engineer.

I can't take credit for this next idea, but now that I've done it, I think it's a must-do when starting a new job. At Lob, every new hire is given a list of people by their manager whom in their first two or three weeks, they have to set up one-on-one meetings with. These are people that are going to be important for the new hire to get to know in their role, or who can provide context or history about the company and business. These can be in-person or occur remotely. (If remote, I would really encourage you to turn your camera on. I think that it is so important to see people's faces and to really feel like they're in that room with you to build that familiarity and rapport).

This was an invaluable exercise for me; it provided me with context on the decisions and directions that had been made before me, and helped me quickly develop relationships and build my network of peers and go-to people for the various things I would need in my job.

Try to distill from others that you talk to what are the biggest problems, pain points, and challenges that they think the organization is facing today? These are good things to take note of so that as you settle into your role, you could try to find ways to improve on these things.

4. Learn in public

While you're getting up to speed on so many things, it's a perfect time to start a “learn in public” culture at your company if it doesn't already have one. Learning in public is where you share what you've learned as you're learning it.

We don't have to hide while spending all night trying to cram on some framework we don't know just so we can log on tomorrow and act like we've been an expert all along. We can be open about where we're at today, and at the same time, we can share the learnings so that others can benefit from what we've discovered. Ask questions in public channels and keep knowledge sharing out of DMs. Show others that they don't have to be embarrassed to ask questions.

In turn, when you learn something new, share it publicly and share often. At Lob, we have a Today-I-Learned channel in Slack, where anyone can share anything in there and the whole engineering organization can see it.

Another great way to share learnings is to demonstrate them. If you've learned how to do something or if someone asks you to show them how to do something, instead of just showing them privately, consider recording a video. Now you can share that video and more people can benefit from that demonstration.

With all of this, try to keep the barrier low. Don't spend a ton of time putting together a perfect dissertation or a superbly polished video. Just hit the record button and start talking. The more you do this, the more others will catch on and not feel intimidated to do it themselves too.

5. Know your learning styles; use them to get to know the landscape more quickly

If you aren’t familiar with learning styles and what yours are, a simple Google search can educate you as well as lead you to a number of quizzes to pinpoint what works best for you for learning and retaining knowledge. Once you know what styles work well for you and which really don't, you should lean on that awareness to really focus your plan for how to get up to speed.

That being said, you should probably do at least a little of each approach, because we shouldn't limit ourselves to just our preferred methods. We need to be flexible and perhaps with practice, styles that haven't been working well for us can be improved.

Visual or Spatial learners do best when looking at visual or spatial representations of information. This could include things such as charts, diagrams, images, and anything that shows ideas in an illustrated way. You may want to lean on using UML diagrams to help you learn more about the platform, interactions between applications, and even within applications. Build your own integration and activity diagrams, or even just flow charts as you trace through things.
Auditory learners do best when hearing information presented to them. Reach out to those people you've been meeting in that network or peer group you've been building and ask them to walk you through some things. Ask people to pair program with you, either on your coding tasks or theirs. (This can be a gift to more junior folks because it gives them an opportunity to teach and explain something that they know to someone else!)
Verbal learners do better when reading and can retain that information better when they write or speak about it. So read the docs! As you learn more, start writing and contributing to those docs. Read code, especially PRs, even if you're not the requested reviewer. This can help you see firsthand where things are being done and how. This can be a great way for verbal learners to quickly pick up languages, frameworks, and libraries they're not familiar with yet by seeing exactly how those technologies are being used.
The kinesthetic learning style is learning by doing, and I've lumped this together with the logical learning style, which involves the use of logical reasoning when processing data and solving problems. In software engineering, I think these two often go together. It may seem like the chicken or the egg situation. If you're trying to learn and get up to speed, but the best way you learn is by doing and solving problems, how can you do things if you still don't know enough? Sometimes you just have to jump into the deep end and be over your head for a little bit. So find projects or tickets to work on that force you to learn your way around.

Last but not least

Try to have fun! Hopefully through all of this, you can keep that Imposter Syndrome monster at bay and really believe that you are exactly where you're supposed to be—and deserve to be, and you can just enjoy the process. Enjoy meeting your new teammates and forging new relationships. Get to know your manager and let them be an ally for you. Build that peer group so you have other people you can confide in and share with if you're feeling stressed, frustrated, or overwhelmed. And celebrate and be proud of your accomplishments—you worked hard to get here and you should feel good about it!

Content adapted from Lob Staff Engineer, Erin Doyle’s, presentation at REFACTR.TECH: How to Hit the Ground Running Starting as a Staff Software Engineer at a New Company.

Stop Wasting Connections, Use HTTP Keep-Alive

Lob Developers — Fri, 04 Nov 2022 16:44:49 +0000

With the proliferation of third-party APIs and microservice architectures, modern web servers can make as many outgoing HTTP requests as the number of incoming HTTP requests they serve. A typical web application can interact with third-party APIs to handle payment processing, send email, track analytics, dispatch text messages, verify mailing addresses, or even deliver physical mail. A server can also rely on internal APIs to fetch account information, start asynchronous processes, or perform complex searches. Programs that initiate a high volume of outgoing HTTP requests must minimize the overhead of each in order to remain performant and optimize resource utilization.

One of the best ways to minimize HTTP overhead is to reuse connections with HTTP Keep-Alive. This feature is commonly enabled by default for many HTTP clients. These clients will maintain a pool of connections—each connection initializes once and handles multiple requests until the connection is closed. Reusing a connection avoids the overhead of making a DNS lookup, establishing a connection, and performing an SSL handshake. However, not all HTTP clients, including the default client of Node.js, enable HTTP Keep-Alive.

One of Lob's backend services is heavily dependent on internal and external APIs to verify addresses, dispatch webhooks, start AWS Lambda executions, and more. This Node.js server has a handful of endpoints that make several outgoing HTTP requests per incoming request. Enabling connection reuse for these outgoing requests led to a 50% increase in maximum inbound request throughput, significantly reduced CPU usage, and lowered response latencies. It also eliminated sporadic DNS lookup errors.

Performance Benefits of HTTP Keep-Alive

Running a benchmark validates the performance benefits of HTTP Keep-alive. The following chart displays the total time taken to make 1000 GET requests for both non-reused and reused connections with a varying number of requests made concurrently. It shows that for all levels of tested concurrency, reusing connections reduces the total run time by a factor of roughly 3.

Another observed benefit of reusing HTTP connections is reduced CPU utilization. On Mac OS X, this reduction manifests in the Node process itself and in a process named mDNSResponder, an operating system service responsible for resolving DNS. Running top -stats pid,command,cpu | grep -E "(mDNSResponder|node)\s" during both benchmarks shows the contrast in CPU usage.

Without Connection Reuse

With Connection Reuse

Inspecting the flamegraph of the benchmark script without connection reuse reveals the reason for increased CPU utilization in Node. A large percentage of CPU time is spent on establishing connections and performing SSL handshakes. For example, the flame fragment below shows that 14% of measured CPU ticks occurred while creating a socket.

It should be noted that initiating connections also incurs overhead for HTTP servers. Therefore, reusing connections also reduces overhead for servers handling these requests.

Flamegraphs of each benchmark are available to explore: flamegraph without connection reuse, flamegraph with connection reuse.

The benchmarking scripts are documented in node-keep-alive-benchmark.

Reduced DNS Errors

Reusing connections also eliminated a set of DNS errors that occurred sporadically within our service. When connections are not reused, a new connection is initialized for each outgoing request. In Node, this initialization includes a DNS lookup to determine the IP of the domain to send the request to. A high volume of DNS lookups can lead to sporadic errors of the form Error: getaddrinfo ENOTFOUND.

Based on several issues in the Node repository (nodejs/node-v0.x-archive#7729, nodejs/node-v0.x-archive#5488, nodejs/node#5436) this error can occur when a DNS server fail to respond, perhaps due to it rate-limiting requests. Reducing DNS lookups can reduce or eliminate these errors.

Tips when Reusing Connections

Check Your Timeouts

In some cases, reusing connections can lead to hard-to-debug issues. Problems can arise when a client assumes that a connection is alive and well, only to discover that, upon sending a request, the server has terminated the connection. In Node, this problem surfaces as an Error: socket hang up.

To mitigate this, check the idle socket timeouts of both the client and the server. This value represents how long a connection will be kept alive when no data is sent or received. Make sure that the idle socket timeout of the client is shorter than that of the server. This should ensure that the client closes a connection before the server, preventing the client from sending a request down an unknowingly dead connection.

Don't Use Node's Default HTTP Agents

For Node services, the agentkeepalive library provides HTTP and HTTPS agents that enable connection reuse by default. These agents also have other sensible defaults that the standard libraries agents do not.

Wrap Up

Connection reuse should provide significant performance improvements to services written in any language that are making numerous outgoing HTTP requests. Some HTTP clients enable this behavior by default, but not all. Some widely used languages and libraries do not enable HTTP Keep-Alive by default, such as Node, so be sure to check the documentation and source code.

More on HTTP Keep-Alive from Mozilla

Code that works vs. Code that is maintainable

Lob Developers — Thu, 13 Oct 2022 15:53:31 +0000

Lose 5 lbs in 5 seconds? Easy! Just cut off your arm.

Most of us are familiar with the concept of SMART goals: Specific, Measurable, Attainable, Realistic, and Timely. You might argue cutting off one’s arm isn’t really realistic, but by the typical SMART definition, it most certainly is. Therein lies the flaw in SMART goals for engineering: I’d argue sustainable needs to be added to the mix. There is a difference between simply achieving your outcome vs achieving it in a sustainable—or when it comes to code, maintainable—way.

Some of our best lessons come from internal Slack rants from our Engineers; this is a good one I shared after diving into old code.

Here's the old version of the code:

While the code does the job it needs to do—in the sense that it's not actively broken—it could've done with a little bit of polish in a few places that would've made a big difference in the readability and maintainability of it.

First problem

The name doesn't really tell me what's happening. getsWhatsRequired is a name that would apply to just about any function that gets and/or formats data. It doesn't tell me why this function lives on its own, nor what it is doing that's particularly useful.

The second problem

There's a whole whack of repetition taking place inside the URIs. This code is repeated 6 times inside this one function. Being repeated 6 times makes it harder to reason about because you either need to read each repetition and decide whether they're doing exactly the same thing OR you end up skipping over it and assuming it does exactly the same thing each time. In this case it is doing the same thing each of the six times, but you have to exert a lot of mental energy just to figure that out.

The third problem

Inside the repeated code there's an intention and goal that's being lost. We're not chopping up strings and adding slashes and stuff just for the heck of it, we're doing it with a specific purpose! But why? From this code, it's not clear at all.

The fourth problem

Those double equals? They're load-bearing! During some minor code cleanup one of my colleagues went and converted them to the === strict equality operator like ESLint suggested because there wasn't any reason to think that this needed to be a loose equality check.

Turns out, the values coming through weren't 1, they were actually "1". That's very different and this caused an incident because we weren't sending any files for a period of time.

There's one of two ways this can be improved on:

If the value coming through is always a string, then use the string representation instead with ===.
If the value is coming through either as a string or as a number and there's a really good reason for that to keep happening, then use the double equals and document this specific behavior so that Future You / your peer knows that you're depending on this very specific behavior.

And the fifth (and by far the nittiest of picks):

If you're going to define the variable and then immediately return it, you can just return that value immediately without assigning it to a variable first. It's a small detail but it makes the function shorter.

Here is the new code

Most of the changes I've made above pretty well to the opportunities I just mentioned, but there's one area I wanna spend a bit more time explaining:

These are the two lines I've introduced in place of the code that was repeated six times.

You could easily argue that these two lines could be one line that reads like this:

That'd be even more space efficient, right? Right! That'd be 100% correct! And here's when context comes into play. You wouldn't know it by reading this code on its own, but all the order assets? They're uploaded into the order assets service, with the folder being the order ID without the prefix.

Knowing that the order ID without prefix has a specific meaning outside of this context, I think it reads better to separate the id prefix slicing from the path composition.

So having criticized this bit of code at length, I want to be real clear about a few things:

It is not the fault of the engineer that this code that was merged in was unpolished. Speaking on behalf of EMs and Staff Engineers, it's our responsibility to set the standards and support our colleagues in writing code that is well polished and maintainable—and we failed to do so here. A valuable lesson learned.

Also, none of this applies to code you're not ready to merge. The code I write when I first start solving something is often very different from the code I end up with. Don't spend time polishing code you'll throw away in 15 minutes.

In response to my ranting, I received the following valuable advice from my colleagues, and I wholeheartedly agree:

“Instead of running into analysis paralysis when I'm trying to figure out the "best" way to write a chunk of code, I'll often just start writing the crappiest brute-fortiest whatever comes to mind first that works. Then I'll write my tests (if I didn't already write them using TDD which I recommend of course). Now that I have a safety net that the code is doing what it needs to, now I can clean it up and feel confident that it still works.” - Staff Software Engineer Erin Doyle

And I’ll leave you with a perfect summary of the above from Engineering Manager Ross Martin:

“Make it work, make it right, make it fast—in that order."

By Lob Staff Engineer Rich Seviora

Tracking changes in your Elixir code with the Audit library

Lob Developers — Tue, 13 Sep 2022 14:33:32 +0000

Part of my work with Lob as a Staff Engineer is to implement various USPS standards pertaining to address autocorrection (aka CASS Cycle ‘N’ and ‘O’). These standards have a lot of subtle details, so the Elixir code that implements them can be intimidating. It is difficult to get this complex code to do what you want, so I started to investigate techniques that would help me track how my changes impacted the results. Before we dive into my solution, let’s first give a quick recap on how functional data structures differ from their imperative counterparts.

A recap on Functional Data Structures

One of the counterintuitive aspects of writing functional code is the constructive nature of data structures. You cannot mutate an existing data structure — if you want to make a change, you must create new elements containing the change that you desire without altering the existing structure. Essentially you make a new version of the data structure that embodies your change that will share common parts of the previous version that stayed the same.

Let’s take the concrete example of a binary tree. The tree, T1, has six nodes: A, B, C, D, E, and F:

Destructive vs constructive binary tree insertion

The traditional imperative way to add G to this tree would be to overwrite the right field of the parent node containing F to point to a newly created node containing G as illustrated by the left side of the diagram. But in a functional language like Elixir, we don’t have the ability to overwrite fields. Instead, we must create a new spine for the tree containing the nodes leading to the newly created G node while sharing the unchanged nodes from the original tree, T1.

At first, this seems terribly inefficient — after all the imperative counterpart just needed to perform one (destructive) update. But it’s not all bad news in the functional world; although in the tree example we create log₂(n) extra nodes, our old version and new version can happily co-exist. In fact, by merely keeping a pointer to old versions of the tree, we can trivially implement an infinite undo facility (space permitting). In the imperative world, this is a bit trickier — we would need to record the changes via a list of deltas and rerun those deltas if we wanted to access an earlier state. In the functional world, all these versions can simultaneously coexist for a small investment in space. Neat! In the imperative world? Not so much…

This also illustrates why it’s easier to reason about functional code versus imperative code — in the functional code, everything stays constant. In the imperative code, a pointer to a structure is no guarantee that structure will remain intact. This is also the reason that imperative concurrent programming is fraught with difficulty whereas functional programming lends itself quite naturally to working in parallel. No need for critical sections, semaphores, monitors, etc. Those constructs are all about protecting some piece of code that side effects from being executed simultaneously in different threads/processes. No side effects — no problem!

Implementing data structures functionally is actually a fascinating area of study — I highly recommend Chris Osaki’s Ph.D. thesis on the topic: Purely Functional Data Structures.

The problem

Now that we’ve refreshed our understanding of functional techniques for implementing data structures, we can move forward with an explanation of the basic idea. The core of our program doing address correction manipulates a very large %Address{} struct. The program works in phases and in each phase a different set of fields are populated with new metadata either deduced from machine learning sub-systems, or extracted from various USPS- supplied metadata. The final Address struct will have the answer we seek (we hope) and a whole lot of additional metadata that might help us understand how it came to that conclusion. But with a data structure as large as Address with over a hundred fields, IO.inspect at some strategic locations isn’t going to cut it — the amount of information printed to the screen is just too overwhelming. We need something a little bit more precise.

Making a difference

A key component of our library is the difference engine. This takes two objects and computes a list of the differences between them which makes it easier for us to visualize the evolution of our data structures over time. First, let’s decide how to represent a difference. A delta has three possibilities:

update, replace one item with another
delete, remove an item
add, add an item

With our representation settled, we need to figure out our implementation strategy. We simultaneously recurse over the two objects we wish to compare until we find a point where they diverge and we return the difference. So our main function will look like this:

This delegates to a three-argument version that takes the current path as the first argument. We flatten the result, reverse the paths (because they’re call stacks) and then order the results by length of path.

So now we tackle the major cases for delta: struct, map, list, tuple, and everything else:

Let’s tackle these one at a time. For structs, if they are the same kind of struct, convert them into a map and carry on; otherwise, they’re already different.

For maps, we want to compute the overlap of keys: a_only, only in a; b_only, only in b; and common, appears in both. With that information, it’s simply a matter of computing the difference of the values of the common keys:

For lists, I take a simplistic approach of zipping them together to compare the elements pairwise. (I think computing the Levenshtein edits would be overkill).

For tuples, if they are the same size convert them to lists and let delta_list process them, otherwise, they’re different.

This just leaves the base case and we’re done.

Ok, I suppose we should define empty?

And that wraps up our description of our difference engine.

Elixir’s macro magic

A lot of folks don’t realize that Elixir is actually quite a small language; most of the features that you know and love are implemented via the powerful macro languages. Don’t believe me? Check out all the language features implemented inside the Kernel module: @variables, def, defdelegate, defexception, defguard, defimpl, defmacro, |> etc., etc. This also explains why the syntax feels is a little clunky in places — it’s likely implemented as a macro!

The basic idea

Our basic idea is to define a macro, audit, which embeds a paper trail inside an audittrail field. It needs to be a macro because we want to capture the file and line number from where the macro was called. Basically, this field will contain a triple of: the last version of the data structure, the filename, and the line number.

Let’s get started by defining some types:

So our audit consists of a snapshot of the struct, a filename, and a line number. It’s considered good macro design practice to isolate as much of the functionality as possible in a function. So here’s our audit_fun:

So now our macro is extremely simple:

To put it to use, we want to be able to generate a changelist given a struct that has our magic audit_trail field:

Now we just need some helper functions to turn that changelist into a human-readable string:

There are two nuances in this code. Firstly, as we’ll be looking up our sources files a lot, it behooves us to cache them. So we implement a simple Agent to accomplish that:

Secondly, by their very nature source files are subject to change. So for this information to be useful at a later point in time, we should use git SHAs to specify the version of the file that we’re looking at. To accomplish this, we have a module to do all the GitHub URL wrangling:

The final result

With all these components in place, we’re finally in a position to demonstrate our library in action.

This is a somewhat trivial example. The ideal use case is for much larger struct where it becomes hard to see the wood from the trees when you IO.inspect results. The use of the difference engine shines in these scenarios.

For extremely large %Address structs in our Address Verification code, this library has saved my bacon on several occasions. I hope you find it equally useful.

Conclusion

We’ve described a simple library to track changes to data structures. If you’d like to kick the tires and try it out, you can find it in hex.pm under audit.

‍

Elixir Streams, Elasticsearch, & AWS S3

Lob Developers — Fri, 09 Sep 2022 14:31:45 +0000

As Staff Engineer on Lob’s Address Verification team, I was recently tasked with standing up an endpoint for customers that supplied them with all the data we had stored for a particular zip code. We use Elasticsearch pretty heavily to store our data so this boiled down to querying ES by zip code and writing the data to Amazon S3. But both services have subtle limitations on how you read/write them: ES caps your result set to 10,000 records, and Amazon S3 insists that you write to your buckets in chunks of at least 4 Megabytes. This turned out to be a perfect use case for Elixir streams!

Elixir Streams

Before we dive into the nitty gritty of Elasticsearch and AWS S3, let’s first do a brief refresher on Elixir Streams.

What is a Stream? A Stream is a composable enumerable that is computed lazily. This is typically accomplished by maintaining a little bit of state that describes where you currently are in your enumerable and a method of computing the next item. A trivial example would be a Stream of natural numbers:

The gory details of how this is implemented under the hood is the topic for another day. You rarely need to get that low level — the Stream library has a wealth of functions similar to Enum that you can use as building blocks to construct your own Streams. And several Elixir library functions natively return streams: IO.stream will turn a file (or other source) into a Stream allowing you to process very large files without having to read them into memory first in their entirety. The lazy aspect of Streams means that the minimum amount of the Stream is computed that’s needed to generate your answer.

Imagine we want to find the 100th natural number that is palindromic and also divisible by 3:

What I find elegant (and efficient) about this solution, is that no number beyond the answer, 20202, will be computed. Neat!

If you’d like to dive in deeper, there’s a great section on the Elixir language website. The API documentation is also chock full of instructive examples.

Elasticsearch

Now onto Elasticsearch. The first thing to know is that ES caps the size of the result set at 10k items. After scrutinizing the documentation, it became clear that the original method that popped up on my Stack Overflow (Scroll API) had been deprecated and the new approved method was to use PointInTime IDs. PIT IDs essentially give you a snapshot of your index at one point in time so that you get a consistent view of your search results across multiple queries. The process is:

acquire a PIT for your index
perform your queries using the PIT (without the index — it’s now implicit in the PID)
delete the PIT

That last step is crucial — PITs are extremely resource hungry so it is vital that you delete them as soon as you’re done. I borked our staging ES in my early experiments with PITs because I wasn’t deleting them and the expiration parameter seemed to have no effect.

The other thing we’d like to do is abstract away the underlying PIT and paging mechanism for the end user so we provide an interface that just takes an ES query and generates a Stream of hits. Let’s start by creating and deleting the PID.

*Creating the PID is just a matter of requesting a PID associated with the index that we’re interested in running queries against.

Deleting a PID is a simple HTTP delete except that the PIDs are huge so you need to supply them in the body and not as URL params. Some HTTP libraries won’t let you do this. The HTTP spec is a little muddy on the matter and last I checked on Stack Overflow there was a healthy debate on the subject. This is why we use HTTPoison — it allows payloads on DELETE requests.

Now that the PIDs are sorted out, our next order of business is figuring out how to leverage them in our queries. Our basic query is enhanced with three extra parameters:

size of 10,000 (the maximum allowed by elastic search)
pit, a hash that contains %{id: <id>, keep_alive: “1m”}
sort, a hash that contains %{_shard_doc: “desc”}

(For sort, you need to provide something and _shard_doc is baked into every Elastic search index so that’s nice and portable.)

With our basic query down, we can focus on how to generate the sequence of queries that will extract all the items that satisfy our search criteria. The trick here is that we feed into the next query the value of our sort field from the last hit of the previous result, like so:

Armed with these two query-generating functions, we can code up Stream.iterator that will pull all the results out of our desired index:

We need a high-level function that takes the results from this function and produces a Stream of hits. But there’s a wrinkle — we have to delete the PIT when we are done with the Stream. But how will we know? The solution is to pass in a consumer that consumes the Stream and then we delete the PIT afterwards, like so:

But if our query returns less than 10k results, we don’t need the whole PID/sort/search_after machinery — one query will do the trick.

Now we need a top-level function that query the size of the result and chooses between stream_one or stream_many depending on the value, like so:

AWS S3

The library we use to access Amazon S3 is ex_aws_s3. It handles all the low-level details for us but it does have one requirement —the input data stream must be chunks of at least 4 Mbytes. To accomplish this, we use the Stream function chunk_while. This takes four inputs:

the enumerable to be chunked
the initial value of the accumulator
the step function
a finalizer function

Our first decision is what should the accumulator look like that gets passed forward by the step function? It obviously should contain the list of the items in the current chunk. But more subtly, it should also contain the size of the current chunk so we don’t waste resources recomputing it each time. So that gives us a two-element tuple containing a list and an integer.

Next, we turn our attention to the step function. It should check if the current size is greater than or equal to the desired chunk size. If it is, we should take the list of items from the accumulator and convert them into a chunk using convert; if not, we should add it to the current chunk (and update the size) using add_chunk.

What should add_chunk do? Just push the item onto the front of a list and increase the value of size by the current chunk’s size.

The behaviour of convert depends on whether we care about the order of the items in the chunk being preserved in the output because the items in the list will be in reverse order so need to be reversed. But if we don’t care, we can skip that transformation. Putting this all together gives us:

We can now combine all the components we have written to produce a simple controller action that requests an output stream of hits from our Elasticsearch section into 4M chunks that will satisfy the requirement of the ExAWS.S3 module:

And that wraps up our exploration of Elixir streams, Elasticsearch, and AWS S3. Hope you enjoyed it!

Webhooks: the Remix ft. Svix

Lob Developers — Tue, 06 Sep 2022 20:33:07 +0000

In chasing our goal to be the world's largest sender of direct mail, our systems need to be robust and scalable; one place where we've made major upgrades to support our next phase of growth is in our webhooks.

Our webhooks product is a labor of love, built from the ground up with our own blood, sweat, and tears. But when just maintaining it was bringing our engineers to tears, it was time to think about an overhaul.

So what are webhooks?

Webhooks are API push notifications, or HTTP requests, that get triggered by specific events and send data to a specified URL. At Lob, we use webhooks to send real-time notifications to our customers based on events within Lob’s system. A common example for our customers is to use webhooks to get notified when a mailpiece is delivered for their tracking and analytics. (Here is a complete list of events triggered by Lob’s webhooks, and if you want to try your hand at integrating, here is a tutorial using Node.js.)

Lob currently sends about 50 million events to our customers each month—needless to say, we need this functionality to be reliable and scalable.

Webhooks OG

Webhooks V1 was built completely internally, and while it’s been performant, it was not an ideal solution long-term for a number of reasons.

An overly-complex design made it difficult for developers to understand and work with.
Not scalable: V1 utilized an AWS service called simple workflow (SWF), which has a hard limit on concurrent executions.
Expensive: The AWS service cost was increasing and impacting our engineering budget.
V1 lacked key features that our customers have requested such as rate limiting, OAuth, easy replay of events (and because of the complex design, these features would be very difficult to add).

Overall, Webhooks V1 was showing its age and giving a poor experience for both Lob engineers (hard to work with, difficult to debug) and the customer (less features, slower incident resolution).

The Remix ft. Svix

The team was faced with the classic dilemma of build vs buy. We’d already tried the build option and wanted to explore the buy option. Our first attempt resulted in accidental complexity. We thought an out-of-the box solution could potentially prevent this. Enter 3rd-party web service: Svix.

Svix is a YC-backed webhooks-as-a-service platform. As Svix notes, “webhooks are harder than they seem” (true statement), so their goal is to reduce engineering time, resources, and ongoing maintenance.

For our webhooks infrastructure, in both versions, events flow from a few sources, like the USPS, and all are routed through Lob’s webhooks REST API. In V1, what followed was a lengthy and complex process—involving multiple queues, workers, and different services—before getting to the customer. In V2, we replaced this complexity with Svix: we now have just one queue, one autoscaling worker, and Svix.

Version 2 includes changes to the Lob dashboard. Requests go from the dashboard to Lob’s API, then to a new webhooks service we built to send the requests onto Svix.

As Svix is a growing startup, there was an initial concern about their ability to handle our scale. As a part of due diligence, we ran a series of load and functional tests with them before deciding to move forward.

“We had a robust service that automatically scales with usage, but we weren’t ready for how quickly Lob’s traffic can scale from normal levels to very high requests-per-second. We had to redesign parts of our system and make our autoscaling much more aggressive to be able to handle these sudden spikes,” said Tom Hacohen, Svix founder and CEO.

This is just one example of how, as we proceeded with implementation (and still now, after launch), it’s been a collaboration. Svix has been able to evolve the product to meet our specific needs. They added servers in our AWS region, rather than have us use their EU-hosted ones. This vastly improved our message processing throughput. They also implemented persistent HTTP connections, which also improved our throughput by preventing unnecessary duplicate authentication requests and TLD handshakes.

The Release

In simplifying our webhooks infrastructure, we achieve greater scalability by eliminating dependencies on some of the individual components (like SWF concurrency limits).
We also made our engineers’ lives much easier. No more endless manual interventions, and late nights trying to debug. Instead of having to deal with seven different repos to make a webhooks change, it's now very simple.
Our costs became more predictable. With V1 our costs scaled as a function of event count. With V2 we have a flatter cost structure and decreasing cost per event as we scale.
The new dashboard is feature-rich: customers can now easily self-serve and replay web attempts themselves; they also can specify rate limits when creating or updating an endpoint.

Webhooks V2 is now a scalable product that is easier to work with, offers more robust functionality, and is budget-friendly.

We were excited to get this project into use by our customers. We rolled out to one of our large enterprise customers, and within the first week and a half, they sent over 10 million webhooks. We've added a few more features to our service, and have asked Svix to do the same; V2 will be GA for all customers in the next few weeks.

Adding a third party to the mix was definitely the right decision for us in this case. In addition to happy customers, our engineers have their evenings back (just in time for the new season of Stranger Things).

This article was adapted from a presentation by engineers Martin Han and Zac Leids.

Why we did a load test in production

Lob Developers — Wed, 31 Aug 2022 19:52:45 +0000

We recently tackled load testing prior to a new product launch — and yeah, we did it in Production. Read on to learn why we’re not totally crazy, and to learn more about load testing and how it can be applied, and adapted, to your use case.

What is a load test?

So what is a load test? It’s an Experiment! When we design systems, we can only do an estimation of how services will handle larger volumes of data. Therefore, the load test is an experiment to determine in reality how our systems handle this surge of data.

Why bother?

We want confidence that our system will scale and handle volume as we expect. Are we scaling up correctly, how do our databases handle this surge of data, how is our application latency affected? Or, we want to walk away with learnings about bottlenecks and/or issues that we may have overlooked in our system.

These are not mutually exclusive. In fact, I would say that the most successful outcome is to achieve both of these goals. Success is success! Even failure can be a success. We should use load tests to find issues before our customers do.

Our use case

Campaigns is the name of the Lob feature that allows for automation of direct mail campaigns, at scale. Those last two words, “at scale” are exactly why we wanted to load test the backend service that serves as the brains behind Campaigns:

It handles the ingestion of an audience file (i.e., some sort of tabulated data, currently a CSV but in the future could be Parquet files, JSON, etc.) This means reading it efficiently and chopping it up into individual rows that can be processed by our API.
It handles the creation of mailpieces. This means taking these individual rows and creating a mailpiece from the details of the CSV.
It handles record keeping. This helps us keep track of mailpieces within the system to surface to our customers as well as help with reconciliation in case something goes wrong.

So…What happens if someone wants to send out millions of postcards? Well, I was challenged to find out. Not gonna lie, it kind of felt like I was getting handed the keys to the Benz.

How do you conduct a load test?

The answer is that it all depends on your service. If you don’t already have a workflow diagram, create one. From there it should be obvious: however your application ingests data, that's your ingress point. One tool we use is k6, an awesome open-source tool that can be used to simulate API request traffic (i.e., simulating traffic for multiple users, making multiple requests per second).

Considerations

We started with a list of questions (and how we applied them); these will differ for your specific workflow, but these should get the wheels turning to direct your own load test.

What is the scale/volume of the test?
For Campaigns, a load test meant uploading a massive CSV. In this case, we used a CSV file with millions of rows, which means we would be creating millions of mail pieces.

Where will you test?
Is your test system a true reflection of production? We ran the load test in production/live mode. Say what? That's right. Lob's API system results in physical output: an API request means that data gets routed through our system and to our partners, who then print a mailpiece that is inducted into the USPS mailstream—that is, an API request typically results in a physical mail arriving in a mailbox. So while we have a “test” system, creating mail in test mode doesn't kick off certain systems that we were hoping to test via the load test. Our production system is "live" in the sense that everything happens for real, so I had to really dive into all the different parts from our system end to end (p&m API, rendering, routing, partner systems) to ensure that even though we're in production, we don’t actually create mailpieces.

When will you test?
We identified the best off-peak period to run this load test, and also took into consideration how long it would take.

What are some downstream systems that will get kicked off by this test? How do we make sure the results from those systems are acceptable?
For us, this meant asking, how can we ensure we’re not billed for any API usage from this test?

What are the warning signs I should be looking out for in our execution services X, Y, Z?
Ex: We kept an eye on the service that creates the PDF from the digital file.

What are the warning signs I should be looking at in our notifications services X, Y, and Z?
We looked at changes in our webhook system throughput compared to expected throughput, as well as watching out for an elevated rate of 5xx errors (e.g. 503's)

How do I monitor the health of our infrastructure to ensure we’re not red-lining any particular resource?

What are your benchmarks/metrics?
Benchmarks/metrics are identified upfront so you know what success/failure looks like. Some key metrics we measure are throughput, latency, and average time to render, for example.

Wrap up

Discovery is half the fun of a load test. And remember, even “failure” is a success. With a list of findings in hand, the team can review and determine exactly where to invest engineering resources to improve the system, i.e., what changes will give you the most bang for your buck?

Two important learnings for us were:

Our previous expectation was that our API workers in charge of creating mailpieces were the bottleneck—this turned out to be wrong! Our bottleneck ended up being our ingestion system, due to the worker being designed to ingest a CSV serially. Ultimately this led us to redesign this worker to ingest concurrently.
We also discovered an unbounded transaction within our code; we were able to rework this particular operation to run in batches.

In addition to meeting the end goals of gaining confidence and identifying opportunities for improvement, teams can learn a lot about internal architecture and workflows—even outside of their own team or area of expertise.

Getting to run this magnitude of a test on our production system was an incredible experience. I previously mentioned doing this load test was like getting handed the keys to the Benz, and I gotta say, it was an awesome ride.

Article adapted from a presentation by Lob Software Engineer Sishaar Rao.

Pivot! From Bootcamp to a Career in Tech

Lob Developers — Mon, 29 Aug 2022 19:27:34 +0000

General Assembly & Girlboss recently hosted a workshop, Learning New Skills is the Key to Career Freedom, “for career changers, mission-driven hustlers and those who want to find their purpose, passion and make an impact!” The session featured graduates of General Assembly (GA) bootcamps sharing their success stories—and offering advice—about shifting to a career in tech.

Career coach Jennifer Harrold dove right in asking probing questions about the most important steps each took initially, and along the way, that resulted in a radical career change. Below are some of the takeaways from these inspiring women who in the moderator’s words, “had no technical skills whatsoever, and yet now here you are sitting at the top.”

First steps

“The most important step for me, that was also the hardest one, was just committing to attend General Assembly,” said Mocha Brown.

“I told myself a lot of lies and made myself insecure about pursuing this career. I thought, ‘I am an artist; I have a creative brain.’ This field felt like the complete opposite of where I should be. I didn’t have a degree, all I had was experience in retail. It was too late for me to start a career in tech. I’d have to have a four-year degree in computer science. But I'm really glad that I didn't listen to those negative thoughts because here I am.”

Today, Mocha brings that creativity to translating complex billing and business rules into Javascript and SQL—as a Software Engineer at Lob. “The first step is letting go of any negative view of yourself or stereotypes about entering tech, and then committing to learning and pushing yourself.”

Kelly Smith, after 5 years in the advertising industry, is now a UX designer at PayPal. “I decided to pivot into UX because I realized that I was miserable at my job,” Smith said. “Acknowledging that life is too short to be miserable, in your job, with any relationship, and just owning that and trying to improve…I felt like I was capable of doing so much more.”

Beth Wolffram was getting her masters degree in Human Rights Law when she realized she wasn’t interested in going into a policy role. “I was more interested in ‘Why are things built the way they are?’ and, ‘How do those decisions get made?’” She acknowledged that GA opened her eyes as to what was possible and could provide her with processes and tools to transition. Beth is now a UX researcher at Google.

She prescribed homework like checking out blogs, websites, and influencers to see what part of UX design sparks your passion, “most UXers have a specialty, but we're also all generalists.” She also recommends doing some coding, to better understand technical underpinnings of the web and how apps work. “It's just taking it one step at a time and not feeling overwhelmed…You are the creator of your own destiny, so places like GA can help you get you where you want to go, but it's really important to be honest with yourself and what are you interested in. What are you curious about? And let that guide you. It will lead you to the people and opportunities that you need.”

As a veteran of the U.S. Navy, Yohlanna Cort not only attended GA’s bootcamp, but also the military's. After she left the military, she felt a bit lost and doubted her exit. But ultimately, decided that going into tech was for her.

“It's just knowing that you want better for yourself, and you feel deep down that there's more for you out there. And so never ever deviate from that, always follow that, no matter what anyone says,” Yohlanna said. “I was working four jobs at the time in Virginia and I really just dropped everything, and some of the family members thought I was crazy, but I knew in my heart that this is something I want to do.” Yohlanna is now a Software Engineer at JPMorgan Chase in NYC.

Bootcamp

Yohlanna said, “I went to college, but when you go out into the ‘real world’ it can feel very different.” On the other hand, “Bootcamp simulates reality for you…Something will go wrong now and I'm just like, ‘Well at least I have a practice for this.'" In addition to white-boarding—and snacks, which sadly are absent from the military's boot camp—Yolanda appreciated learning how to utilize resources like Stack Overview or YouTube. She credits bootcamp for her ability to apply her newfound understanding and appreciation of her learning style to her working style. "[GA] gave me a starter toolset and now it's on me to acquire other tools and sharpen the ones that I do have.”

“GA completely prepares you by going through the design process at least four or five times in groups and individually, and literally that's what we do in the workplace,” said Beth.

Kelly also noted that her assignments at GA mimicked the real workflow of a UX professional. For example, “We would conduct user research by just going up to strangers and talking to them, and I was terrified, but that was just part of the job…to be a UX designer you have to be curious…that took me a long time to get over but it's something that has carried me throughout my career.” Kelly credits bootcamp with the lesson, “You have to get comfortable with being uncomfortable.”

Each of the women agreed that GA was hard work. “Stick to it! Even though you may not understand everything right at the beginning, that's normal,” Mocha said. “If I had given up or stuck with the assumption that because I had no idea what I was doing I couldn't be successful, then I wouldn't be here today.”

Life after Bootcamp

Beth noted GA’s focus on adopting a growth mindset was key, and she took this out into the workplace. “Overall, I think that's probably the biggest value that GA delivers because on top of the skills, just being completely equipped to be confident to go into that first role is huge. You know that everyone's got your back—your coaches, fellow students, and your network and everyone you've met—you're not out there alone anymore,” said Beth.

image courtesy of https://www.techtello.com/fixed-mindset-vs-growth-mindset/

GA’s coaches reinforced the idea that applying for jobs is more than as Mocha puts it, “Hey, I can code, hire me.”

“I think one of the most important things was learning how to brand myself because outside of the technical skills, companies want to know what it's like for you to work with them. You can do that by branding yourself on your resume and on your LinkedIn profile,” Mocha said. “Also, it's a lot of work, but for each job that I applied to, I made sure to showcase the specific skills that they were asking for in the job description.”

As far as the interview process goes, Mocha learned being positive is really important, but so is preparedness. She offered the following advice: “If you are going to interview for a company that may give you an algorithm problem, the best thing you can do is go to Hack Ranker and practice one or two problems a day.”

“One of the biggest things that helped me land a job thanks was getting paired up with a mentor—another GA grad—who gave me that boost of confidence that I needed and helped me refine my portfolio,” Kelly said. Speaking of your portfolio, Kelly recommends you memorize your projects inside and out in order to comfortably and confidently talk about them—this will prevent you from just staring at your screen and scrolling, and instead, allow you to really engage with your interviewer.

“There were moments after [bootcamp] graduation where I had imposter syndrome—which I think is normal when you're changing careers—but I just felt proud of myself for doing something so scary, completely pivoting, that it gave me confidence,” Kelly said.

When facing the fear of “only” being a bootcamp grad compared to someone who has a four-year degree, Kelly reminded the group, “they asked to talk to you so they're interested.”

It’s a message worth repeating and Jennifer thought so too. “There is really not a universe in which a recruiter or a hiring manager is going to reach out to you ‘just for Skittles.’ No one has time to pretend to interview you for something they don’t think you can do!”

Beth agreed: “You did it, you've earned it, you do belong there.”

Fear: Then and Now

In twelve months, Mocha went from a conversation she had with a customer about coding (“what is that?”) to getting her first engineering job. During the entire process, the fear—of failing and succeeding—was a constant. “I didn't conquer my fears before joining GA, I learned how to work alongside them. If I had waited until I conquered my fears, I wouldn't have gone through with it,” Mocha said. “It's hard work, but I want to say again, don't underestimate yourself because two years ago there was no reason for me to think that I would be here. I want people to know that it's definitely possible.”

“Feel the fear and do it anyway. It's more important for you to learn and to become who you feel you're destined to be than it is to feel that moment of frustration or uncertainty. I love the quote ‘I didn’t come this far to only come this far’,” said Yohlanna. “Once you're in it and you're going through the frustrating times, you keep going because until you get the life that you really envisioned for yourself, just keep going. You didn’t come this far to only come this far. Please keep going.”

General Assembly offers part-time programming to help working professionals level up through in-depth training in digital marketing, data analytics, product management, UX Design and other in-demand skills. The immersive programming prepares adults for a full career change and entry into roles in Software Engineering, UX Design or Data Science. Get matched with the perfect program.

Girlboss exists for women to (re)define success on their own terms. They are a community of women who inform, entertain, and inspire action through content and experiences such as the GirlBoss Podcast, Blog, and Job board. “We are unapologetic in our beliefs and values of supporting girls and women who are chasing dreams both big and small in a shame-free zone.”

Swipe Right on Redshift

Lob Developers — Thu, 25 Aug 2022 21:16:00 +0000

As a “top pick” in the direct mail automation space, Lob prints and sends millions of mailpieces each year. As such, it's been Lob's commitment from the get-go to minimize our footprint and build a more sustainable approach to direct mail. As a part of this initiative, in addition to ensuring we use responsibly sourced raw materials, we plant two trees for every one that we use in the production of our mailpieces, through our partnership with Eden Reforestation Projects. (For more on our partnership check out this blog post/video: “The Power of Trees: Partnering for Positive Environmental and Community Impact.”)

As awesome as that is, this is not a sustainability story, it’s about the technology we use to accomplish this and why we shifted (excuse the pun) tools.

During a hackathon in 2017, our Engineering team developed automation that runs every month. This involves a Postgres query that pulls how much mail (letters, checks, postcards, etc.) Lob sent the previous month, a calculation the equivalent in paper, which is then translated into a number of trees. (Then we offset our carbon footprint by sponsoring Eden Reforestation to plant double that number of trees.)

Snippet of query:

The result is a summary communication sent via Slack to the community and stakeholders.

Unfortunately, as of late, the automation was pretty flaky: often the query would time out requiring an engineer to run it again in off-peak hours when the Postgres database was under less load; this is not a scalable solution and was yielding less and less success.

Since the query looked fine (accurate), an initial idea was to break it out into separate queries and then combine the results. But this and some other brainstorms just seemed like a hack; sure they would fix the current problem but eventually, those would get big enough to timeout too. So, Platform Engineer Elijah Voigt turned to the rubber ducky channel.

Enter Jack Meussdorfer from the Data team to the rescue.

His keen sense of smell indicated this was a too much/big data problem. He suggested instead of running the query against PostgreSQL, we run it instead against Amazon Redshift, which the Data team already uses for similar data pipelines.

In short, Postgres is a transactional db, whereas Redshift is specifically designed for large-scale data storage and analysis. Because Redshift stores data in a different way—it uses column stores instead of row stores—there is less overhead on queries spanning a large selection of data. (This also means differences in how constraints and indexes are implemented.) Redshift also has a higher capability of processing large amounts of data because it runs massively parallel processing (MPP). If you want a deep dive check out this article Redshift vs. Postgres: Detailed Comparison of Performance and Functionality.)

Is that really the right tool for the task?

Since the necessary data already exists in Redshift, Jack suggested we:

Update the query to be compatible with Redshift (alter the query syntax and add the proper schema/tables) and then
Point the Reforestation query at Redshift.

Boom!** Lifting and shifting the data source from Postgres to Redshift solved the problem immediately: the query went from timing out after 1 hour to literally taking 40 seconds.**

The takeaway? Rubber ducky rules. And, it’s important to evaluate whether you are using the right tool for the job at hand. Postgres was the right tool for the job in 2017, but our data changed—which means our problem changed. While we are big fans of Postgres, our automation needed to scale to our new needs, so for this important query, it’s Redshift FTW.