 we began to see cracks with our current cloud provider, 

Issues in the application layer aside, we knew digging into performance updates in the application could could result in the added scale we'd need as we continued onboarding larger and larger clients.

We however, could not code-optimize ourselves out of Heroku's very costly pricing structure.

So on top of being on the most expensive Heroku plan at a little more than ~$2500/mo, and seeing the road ahead requiring much more scale, the decision was made to graduate to AWS.

I want to use this space to outline what exactly we were paying for on Heroku, delve into the infrastructure I eventually set up on AWS. This by far is not an entirely optimal set up, but was optimized for speed of execution.

From start to finish, the process cost us roughly 2 weeks of development time.

What went into that Heroku cost exactly? Here's a breakdown:

$2000 - 4x Performance L Dynos set to auto-scale to 10 (Heroku talk for compute resources, maximum spend of $500*10 dynos $5000/mo)

I'm fully aware we were using far more compute resources that were necessary, but again a through line here is our team needs to prioritize development speed. We probably could've gotten away with Heroku for much longer had we sat down and optimized our code, but frankly as a pre-PMF company it's not a good use of time.

We chose a very standard AWS infrastructure based on ECS. The full list of services used is below:

AWS Secrets Manager for environment variables

ECS clusters and services to host our main backend app, and workers

CodePipeline for automated deploys on pushes to Github

Distributed Load Testing AWS CloudFormation template

I know some folks swear by Terraform or CloudFormation templates, but we wanted to get the migration done ASAP, and I saw trying to grok the config languages more trouble than using the AWS console. All the infrastructure above was created and setup using the AWS console, and took very little time to scaffold up.

We had a lot of success using NextJS in our frontend app and deploying on Vercel that we wanted to continue using serverless on our backend if possible. We considered Elastic Beanstalk, but coming from Heroku we know we liked the ease of deployment, but we wanted a bit more control over our infrastructure.

One thing Heroku had built out amazingly well was quality of life around deploy triggers from pushes to Github branches, and monitoring. We wanted to fully replicate the ease of pushing to our branches to trigger deploys so CodePipeline was a must for this implementation. Load testing.

This is an amazing tool AWS has developed, their 

 CloudFormation Template. We were able to get the tool up and running using their prebuilt template, record network requests made on page load using 

, and get operational with load testing before release in a matter of hours. More on this below.

Dockerizing the backend. The biggest blocker to getting our backend migrated into AWS was our backend was not serverless. We had a monolith JavaScript app running our GraphQL server, and one additional worker process. I wanted an easy deploy process on AWS so the first order of business was to dockerize our backend services. All of our code lives in a single NodeJS-TypeScript monorepo, so we have two separate Dockerfiles to build our web and worker images:

The two dockerfiles for the apps load and run different NodeJS start commands for our two core processes:

We use objection/knex as our ORM in our backend app. Heroku made it extremely easy to run these knex migrations using their heroku run commands. As we were going serverless, we needed a way to run the migration node command yarn knex migrate:latest/rollback remotely. We added a protected endpoint to run these migrations manually via API endpoint, as I didn't see an immediate way to run a node command against an ECS service.

With our infrastructure created, backend dockerized, and migrations handled, by and large we were ready to move over to AWS. We started moving everything over on our staging environments first, ran our load tests, migrated all data over to our production instances, ran load tests, then pointed our frontend to the new AWS infrastructure.

For both our staging and production data we had to dump and restore our PostgresDB data, and our Redis cache. On staging this wasn't an issue as usage was just internal, so we ran the Postgres commands to dump the database, and restore the data on RDS, repeated the process for Redis.

Using the AWS tool previously mentioned, we ran our load testing tool against our new staging infrastructure with auto scaling set to what we'd use in production, and we successfully handled 3x the largest spike of traffic we had ever seen on our site! Great news. At this point we felt confident to migrate over our backend fully to AWS.

Production was tricky as users were live on the site mutating data. To solve this issue, we temporarily shut down our services with a maintenance mode landing page until our Google Analytics reported 0 active users, ran the database dump, and restored it to RDS, repeated this for our Redis cache.

At this point, all the production infrastructure was in place:

Load balancing and ECS was setup with enough compute resourcing and auto scale settings

As soon as the data dump finished restoring on AWS, we rebuilt our frontend on Vercel with the backend API URL pointing to our new AWS infrastructure, and disabled the maintenance mode from the site!

Once we migrated our infrastructure over to AWS, our monthly estimated cost was going to run us roughly ~$400/mo. This was amazing to see as we saw with our load testing we were able to handle 3x the volume of traffic at essentially 1/4 of the price. But, we were even luckier.

When we incorporated our business using Stripe Atlas, we had been given a $10,000 credit for AWS that we never cashed in. Now that the bulk of our infrastructure was on AWS, it made sense to finaly activate those credits. 0-dollar backend server costs since.

We should've made the move much sooner, but with all things in building a startup, that switching cost came at a cost we never were quite willing to bear until a client relationship was at stake. I could've gone much deeper into the nitty gritty of how everything was glued together, but I hope this gives an over-arching picture of how we migrated for other folks in similar positions.

Platform Migration to Unlock 10x Cost Savings and 10x Scale

I wanted to take this opportunity to briefly show case our process for getting software projects out the door and into the hands of excited users:

Scoping out product and technical features

We'll walk through an example of a product feature developed for 

At Legaci, we always start from the foundational problem we built our company on: as an artist, I feel like I don't own my audience and can't reliably monetize outside of tours and merch.

We typically find a few sources from where product ideas can be sourced from.

Opportunities found from having conversations with our artists

Shower thoughts when reflecting on our product

Prioritization of these is so heavily centered on what the business objectives are for a given stage. For a Pre-PMF company such as Legaci, more weight might be placed on iterating features that'll have an outsized impact on word of mouth growth versus monetization.

Wherever the team lands on choosing a particular product feature to develop, given enough time, the bottleneck typically isn't in the idea phase. It is however, important to understand where these next iterations come from as this may inform certain aspects of the feature to focus in on.

The decision to build out a paid membership feature came from two places:

2. Every creator platform and increasingly social media sites have started introducing the feature set.

Time is the most precious resource for a pre-PMF team, and so we owe it to ourselves to gain confidence on our spread of ideas to best understand what's going to move the needle the most.

There's a lot of established frameworks to accomplish this like 

, but often we feel they're a bit too heavy on the process our team of 3.

The justification to dedicate ourselves to paid memberships came from a more gut reaction. A lot of finding product market fit is "you'll know it when you see it". As we're operating this business, we started to build a sixth sense for when something is working and we're headed the right direction. Don't shy away from this!

A gut reaction is almost a hack to tap into a meta-systematic view of your business, encapsulating all your customer interviews, analytics data, personal creativity, etc.

Product requirements doc and the tech spec

The core question now becomes "How do we make this a reality for Legaci?".

We start with what most would generally call a product requirements doc (PRD), then turn that into a technician spec detailing the blueprint for implementation.

Let's start with the PRD. These usually take the following form:

Fine grained product changes (user stories in agile)

We use this section to back our gut reactions with those relevant customer interviews, analytics data, or any other tangible evidence.

Here we describe from a 10,000 ft view what the solution needs to be. In the case of paid memberships, this was as simple as "clone Patreon, but without the exclusive member posts".

This will take the form of a todo list for where updates need to be made in the app. If needed, this is where we'll add in designs and high level wireframes to mock out flows.

So for paid memberships we would have tasks like "ability to create a membership tier, and adjust the pricing", or "ability to join/cancel/resubscribe to an artists premium membership".

More clarity in the PRD process has an incredible effect in making development easier. Small example, but if a feature can be walked back and isn't critical to MVP, we can avoid adding a new table and all the work involved in that effort.

We break out our tech specs into three or four general sections:

For each product update, we'll go through each of these sections and enumerate what is required. I find it easier to rationalize what the new user experience needs to be first, then work my way backward deeper into the tech stack. But that may just be a personal preference given my background doing years of front end development!

Paid memberships was a huge feature requiring updates in all categories. We needed new views and UIs to allow artist to create their membership tiers, connect their bank accounts, handle payouts, among more. Users also needed to manage all their subscriptions, update their subscription settings, see they're premium members in the community, etc.

Once the front end changes were scoped out, I can begin to think about what new GraphQL queries and endpoints will be needed to support them.

This leads naturally to thinking about new columns and tables needed to store the new data requirements, tables to record artist membership tiers, user subscriptions, etc.

And to support the payments functionality, we leveraged Stripe's Connect platform. The tech spec allows us the flexibility to change our tech spec to fit the integration as needed.

After all the work above is done, writing the code ends up being the easiest part. Actually writing the code becomes a very simple task of going through the tech spec and ticking off checkboxes. This is where you can confidently say you built a massive feature "in a weekend" part in the headline comes in, all the requisite queries and mutations were implemented in roughly 3/4 days.

Since we've parsed out all the work ahead of time, it became much easier to spot clean divisions in tracks of work to split with my cofounder to get this out the door quickly.

For Legaci, TDD is not appropriate. For where we're at, the process is just not quite fast enough. Once the general implementation of functionality is done, then we layer on integration and maybe end to end tests to validate behavior of the code.

We move the code up to our staging environment, which we've been able to keep as close to 1:1 with production as possible. Here we now create a QA checklist using the PRD and perform manual QA. All of the team is involved in QA.

Once we feel good about the changes, and have QAd the functionality, we release our front end and back end apps and monitor for issues.

Our monitoring suite is very simple, we have a few tools to validate the health of our app in production, a quick run through:

 for logging. Dashboards for 400/500 status codes and our app errors are essential.

AWS CloudWatch dashboards. We track memory, CPU, and containers deployed across our ECS services

 to take a pulse check on user experience

Generally we follow this process for more intensive projects with a lot of moving pieces.

One of the many silver lining of working on your own startup is we can quickly adjust to forgoing certain steps if we feel they're not needed or will hamper us in delivering value to our artists.

Having some process to fall back helps immensely in making some sense between the sheer number of things you can be working on as a business owner.


Delivering High Quality Code - Our Process

Lots of businesses benefit from implementing their own loyalty programs. Airlines, restaurants, coffee shops all have some form of loyalty programs to entice recurring visits from the same customers.

Implementing a loyalty program for your specific business however, is a weird, nebulous problem that never seems to quite fit into an existing software offering.

We ran into this while building Legaci. We felt a lot of artists lacked any meaningful way to incentivize bigger commitments from their fans beyond streaming them on 

We wanted to integrate a loyalty program where fans get rewarded for listening and watching an artist’s content, then use those points to redeem merch, pre-sale tickets, video shout-outs, etc.

We felt adding a livestream experience tackles two birds with one stone: offer engaging, long form content to users, and lay the foundation for the core “gameplay” of the loyalty program.

For loyalty programs to work, users need to take actions they’re already accustomed to doing. Starbucks customers purchase coffee, airline customers purchase tickets. For artists, their fans are used to listening to music, and watching music videos and other video content featuring their favorite artists.

 ecosystem alongside our loyalty program, we decided the best way to deliver something was to take advantage of our artists' catalog of music videos and content on YouTube.

So, we created a "livestream" where fans watch a playlist of YouTube videos curated by the artist. The videos remain in sync across all users so the fans are always watching the same content. For every minute a fan watched, they earned 1 point which they could later redeem for prizes.

Because we’d be transacting potentially hundreds of thousands of requests a minute, we wanted this system to have as few moving parts as we could afford.

SQL table acting as a ledger for the individual records of points awarded

SQL roll up table storing the summarized points for a user

Redis queue to manage recording transactions, and preventing double spend and bad actors

Few reasons why these were the choices we went with.

The SQL tables are very simple, we track the user ID, artist ID, and amount of points transacted. The rollup table was necessary to cut back on unnecessary, and repeated calling of the main ledger to summarize the latest point values for a particular user.

We needed still to maintain a ledger of all the point additions and subtractions to make sure we have a high level of confidence in the system. This is extremely important as these points would be used by fans to unlock access to extremely valuable products. Maintaining a ledger let us confidently report back to our users, and artists that the points being represented were indeed correct.

Leveraging Redis’s single-threaded server, and a basic system of locks cuts back entirely on bad actors attempting to take advantage of the points system. Logging in on different tabs, different devices, and the like are all attack vectors to gain the system. Again on the issue of increasing confidence, these were all necessary requirements we had to account for.

We can cleanly split this up into two sections: the frontend and the backend code.

The frontend is a very simple layer, that calls a GraphQL increment endpoint once every minute, granted a few conditions are met (the user is logged in, is a member of the fan club, has not gone past the daily allotment of points).

The backend becomes very simple once we make the right infrastructure choices. Broadly, the increment endpoint enqueues a job indicating this particular user for this artist needs a point added or subtracted. We also track for what particular instance this user is transacting points, whether that's for redeeming a prize, or watching TV, or something else.

All the verifications, checks and processing is done in a serverless function that verifies whether or not this user can have this transaction recorded.

Processing function that verifies whether or not this user can have this transaction recorded:

Create a new entry for this transaction, or update an existing one if it already exists:

Finally, we update the rollup table to reflect the latest point values for this user, and trigger a Mixpanel event for general observability of usage in the app.

The hardest challenge in rolling your own loyalty program is figuring out that core gameplay your users/customers/fans will engage in to get points. This needs to be something that can be done solo, is existing and familiar behavior, and is engaging.

Once this is hashed out, implementing the core system isn’t too esoteric of a software engineering problem. The main issue is understanding where the vectors of abuse are in your particular game, and clamping down on those with good architecture, and checks along the way.

Building an In House Loyalty Program

 were getting to be very React-component/Virtual DOM heavy. I'm sure rendering large, complicated feed pages is a Web 2.0 solved problem given the likes of Reddit/FB/Twitter/etc. However, given we're a pre-seed company with two technical cofounders, we're all we had to get this feature working well for our users.

Our feed pages are pretty typical from other social media platforms, users can upload native photo/video content, link out to Twitter/IG/TikTok posts, and drop text-only posts.

The issue we were running into was certain pages with a lot of activity would be unusable as hundreds of asset heavy components were rendering in at the same time.

Below are some techniques I spent roughly a week or two on, diving in and making quick changes to improve performance significantly.

Use the buttons in the top right corner to record a page load or transition. What’s presented is a list of all the components rendered. The more orange a component, the more longer it took to load in this commit. The timing for a component such as BrandPosts (1.1ms of 633.7ms) means BrandPosts and its children collectively took 633.7ms to load, but BrandPosts itself only took 1.1ms. Dive into why BrandPosts is taking so long to render by repeating the process in child components, look for anomalies or refactors.

Similar to the React profiler, this tool also shows time for components to render, but gives a bit more insight.

Dive into the function call stacks and see why certain files are taking time a long time to load:

Run in your terminal to find dependencies no longer being used in your project:

A good discussion about unused dependencies can be found here.

Related - we can get into the habit of checking bundle sizes going forward. To find a bundle's size: 

The above methods are what really chunked away slowness on our feed pages. Below are a few techniques I used that seemed correct in theory, but I wasn't able to determine a significant ROI:

React.memo(…) - another option for memoizing components.

Look for unnecessary component renders - leverage use effects and watch variables that may be volatile.

Reduce the overall number of components - paginate data sources

react-window - a good option, but trying to understand fixed heights and widths for our feed components was requiring far too many brain cycles.

Platform Migration to Unlock 10x Cost Savings and 10x Scale

Delivering High Quality Code - Our Process

Building an In House Loyalty Program

Optimizing Performance for ++React Virtual DOM Elements