10 Rules For Building Better Internet Applications

blog_headerIf you’re working on an application that’s been around for a while, has lots of usage and several different teams working on it, there’s a good chance it’s a dumpster fire. It happens to the best of us. Building software with real customer adoption is messy business. While there is no one size fits all, these are my opinions on building more reliable, scalable, and maintainable Internet applications.

1. Go All In On One Cloud

Pick the leading cloud provider that makes the most sense for your team and go all in. Don’t miss half the benefit of the cloud by limiting yourself to the lowest common denominator between cloud providers, which is basically just VMs. This is folly anyway as it is nearly impossible to build an application natively in the cloud without tentacles into proprietary resources from that provider. Trying to be on the light end of the lock in spectrum is not worth the trade off you’d be making in lost productivity and reliability.

2. Forget About Servers

Use containers and/or serverless for 100% of your dev, stage, and prod environments. There is almost no reason to use VMs directly for any new development. While containers and serverless have their own set of challenges they are an order of magnitude better than using configuration management (Ansible, Chef, Puppet) and are your only hope for getting your dev team to participate in DevOps, which is necessary to scale development of a cloud native application.

3. Commit Wholly To Infrastructure As Code

Don’t take shortcuts that question your ability to stand up the application from source in the future. Make defining your infrastructure as code a requirement, not a nice to have. This applies to your infrastructure primitives (e.g. building blocks directly in your AWS account) as well as any third party tools or services you may want to integrate with. If it can’t be brought to life entirely from source code, you should seriously consider an alternative that can. Your application code needs a specifically configured environment to run in. Setting up that environment by hand is like writing a program, deploying it to production, then deleting the source code. Let’s not even talk about that install guide document you pretend is accurate and up-to-date.

Your first line of code should be your Dockerfile, or maybe your serverless yaml or your Terraform. The point is, start with your infrastructure code, at least enough to define your immediate development environment, which becomes the blueprint for production. Store your infrastructure code with your app code, lock step with the application in the same git repo with each microservice. You should be able to clone a git repo from any computer, build the Docker image and immediately start working on it, no matter what language it’s written in or dependencies it has. “It works on my machine” is the joke of the last generation of software development. Think of your machine as a text editor and a Docker engine.

4. Embrace The Ephemeral

I think it’s helpful to think of your code running on a “server” that didn’t exist 5 minutes ago and won’t exist 5 minutes from now (or maybe seconds, or milliseconds). If you haven’t noticed, in the cloud they aren’t even called servers anymore, they’re instances, hinting at their proper use lifespan.

Never write out any state to local disk volumes. Pretend your environment doesn’t even have a local disk. Expect instances of your program to terminate abruptly due to auto-scaling or chaos experiments. Your operations need to be idempotent, and anticipate retries. You need good logging and usage metrics, which your application must be deliberately broadcasting. Throw away your ssh keys and keep the telemetry mindset of your application running on an inaccessible remote location.

This forces the issue of immutable infrastructure. It makes no sense to modify or “patch” something that has a lifespan of milliseconds to minutes. Change is always by terminating the existing resource and replacing it with a new one with new settings.

5. Use Managed Services Wherever Possible

Building and operating clusters of anything isn’t providing business value or helping you to quickly iterate. Wherever possible, skip all that undifferentiated heavy lifting and use a managed service.

Installing a database server for example, on a single server for development isn’t so hard, but doing it in production in a way that is highly available is another story entirely. You have to setup an auto-scaling configuration where nodes can launch and rejoin the cluster, setup and test failover and restore for your HA configuration, and setup and test all the associated monitoring and alarms. It’s easy to get something wrong here and since you’re usually in the data layer, these are costly mistakes. And since our dev and stage environments need to match our production environment exactly (apart from instance size), the easy single server installation isn’t acceptable in any environment. Maybe this would be manageable if we were doing this once for one huge database server, but in a microservices world each with their own isolated data layer, it’s just not practical to self manage with dozens of services. Recalling the earlier rule about bringing everything to life entirely from infrastructure code, usually the only way to accomplish this without a herculean effort is to use a managed service. If you’re insisting on installing everything on EC2 yourself you are missing half the benefit of the cloud.

6. Testing Should Make You Go Faster Not Slower

Testing should give you rails so you can speed, not gates to force you to come to a stop. Your tests (all of them – unit, integration, load, front end) should be a completely automated part of your build and deploy pipeline, and like your infrastructure code, should reside in the same git repo as the component they are testing. The same integration tests should run against all environments, especially production, which is the most important environment to test because it is the only one your customers actually see. This means you need to use test accounts in your real databases, your testing can’t rely on starting from a fresh database each run, which isn’t feasible in production, and demands exclusive use of an environment for testing, which obviously doesn’t scale. This requires a close partnership with development, it simply won’t happen if you treat QA as a siloed activity.

Testing should share the microservices mindset – when we replace a pipe in the basement, we don’t always need to reinspect the whole house including the windows in the attic. Testing is critical, but we don’t want testing at any cost. Don’t let outdated testing practices be the weak link in your agile process.

7. Always Be Deploying

Because I think this one is so important, I’m going to borrow the sentiment from the infamous Glengarry Glen Ross, “Because only one thing counts in this life. Get [the code deployed to production]! You hear me you ***** *****! A, B, [D]. A – always, B – be, [D] – [deploying]. Always be [deploying]. Always be [deploying]!”

The more time that passes between deployments, the higher your integration risk, and the greater chance you are blocking another service which you are a dependency for. This is exactly why we do CI/CD in development. While I’m not saying you need to do continuous deployment in production, the same risks accumulate as time goes on so you want to keep the change delta as low as possible. Plan on deploying every sprint. Most deployments are back-end services with minimal to no customer facing aspects, so this doesn’t necessarily mean visible UI changes all the time.

Each microservice should be independently testable and deployable. When working on a feature that spans multiple microservices, deploy each one as soon as it is ready, don’t squirrel them away to deploy all at once in some blaze of glory.

So put that coffee down, coffee is for deployers!

8. Write Your README Before Writing Any Code

As the saying goes, “If you don’t know where you’re going, any road will take you there.” I love the Amazonian practice of starting a new project or feature by first writing a press release announcing it. I think that causes you to think bigger and to go into the work with a more clear picture of the most important parts of what you’re building. Perhaps the microservice equivalent of that is writing your README first. Give a couple sentences about what the thing does. Show usage examples demonstrating the main interfaces. Show examples of how to build, test, and deploy it. Do this first, in the README in your git repo, not last in some external document management system.

And for the love of God, before any code is written, think about how you’re naming things. Nothing shows lack of thought about how your service fits into the bigger picture of your ecosystem of services like giving it a vague or misleading name. Renaming things seldom happens, or is really expensive to do later.

9. Open Source From The Beginning

Even if you don’t want third party contributions, I think open sourcing your project reinforces a few helpful perspectives. First you are forced to handle secrets appropriately, which makes it a lot easier to get a short term consultant involved, because you can actually share your code base without worrying about changing the secrets later (let’s be honest, doesn’t always happen). Open sourcing your project may also inspire confidence from your customers about the longevity of your platform and inspire the best from your developers who will hold themselves to a higher standard knowing their work will be on display.

More importantly, it serves as a constant reminder that while it takes valuable skill to produce the source code, it has no intrinsic value and can be easily duplicated. I know this is a difficult pill to swallow when you’re spending boatloads of money on development but it’s the truth. Your value is in your data set, relationships, reputation, and execution.

At least look to open source components of your application which may be general use tools, or with a small amount of additional thought could have a general use interface rather than being specific to your application.

10. Iterate on Customer Experience

Andy Jassy ended his 2018 AWS re:Invent keynote with some remarks which encapsulates what this is all about.

“For any of us who are trying to build long standing sustainable businesses, which is hard, the most important thing by far is to listen really carefully to what your customers want from you and how they want the experience to improve, and then to be able to experiment and innovate at a really rapid clip to keep improving the customer experience. That is the only way that any of us will be able to survive over a long period of time. That’s what you should care about, giving your builders the most capable platform and allow them to keep iterating on that customer experience.”

While of course he’s suggesting that AWS specifically be the foundation of that platform, the sentiment rings true, and that’s what all of this is ultimately about, investing our resources iterating on customer experience, and not get bogged down with boring IT stuff.

Super Powers: AWS re:Invent 2016

Amazon knows how to throw a party. They hired the best DJ in the world, literally – Martin Garrix. The re:Invent Party was an experience I’ll never forget. Walking in, you are handed an activity map, showing the locations of the digital lounge, beer garden, dance floor, and play zone including a laser dome, ball pit, bubble soccer, climbing wall, and a labyrinth. But once I got to the DJ area, I was mesmerized.  I stood in the crowd and soaked in the music, the lights, and the special effects. It was truly a performance. Even though I traveled alone and didn’t know anyone there, I felt connected, surrounded by thousands of other like minded people celebrating at this amazing event.  AWS CEO Andy Jassy captured the sentiment best saying simply “it is awesome to be here.”

The conference was sold out at 32,000 attendees and occupied most of the Venetian and the Mirage resorts.  Everywhere you went it seemed like Las Vegas was invaded by swarms of cloud enthusiasts all wearing the black and blue AWS hoodie given to us on our first day.  Fredric Paul from New Relic put it well saying “The whole event has the palpable buzz of a celebration of a successful, growing industry that feels like it’s on the right side of history.”

The theme of Andy Jassy’s keynote was that AWS gives you super powers, and I think it’s true. You have access to essentially infinite resources. As a single developer you can build something over the weekend that you can deploy globally, that can auto scale to handle any amount of volume, store any amount of data, and initially cost next to nothing.  Want to cut your teeth on a 128 CPU server with 2TB of memory?  Spin it up right now, it’s yours for $13.34 per hour.  Where else would most of us ever even have access to a machine like that?  How about a fleet of them on the spot market for up to 90% off?  Need to accumulate 100GB of storage per day?  No problem, write it out to S3 for just $0.03 per GB per month, and know you’ll never run out of storage space.  Werner Vogels, CTO of AWS put it this way – “One of the things the cloud has done is democratize access to technology power.” Just knowing that I can affordably access any resources I need challenges me to tackle something of scale.

Better yet, as Jeff Lawson, CEO of Twilio, astutely pointed out, “you don’t get points for using servers, you only get points for serving users”.  With serverless architecture, you can put together entire solutions on inherently scalable building blocks that can serve millions of users, but only cost as much as a Starbucks coffee until it starts to get some traction. The cloud really has given us super powers.

Jassy wrapped up his keynote with a few words on the next 10 years. Because AWS has grown so substantially in its first 10 years, would the the next 10 will have less invention? “I believe the next 10 years will have markedly more innovation,” Jassy said.  Apparently, the cloud is just getting started.  Instead of debating wheather companies would use the cloud or not, Jassy said “They will spend all their time using the cloud and its massive capabilities to build new business that we never imagined before.” He continued saying “This is a world that I want to live in. And this is a world that I think is very exciting to everyone in this room.” I had a blast at re:Invent, and left pumped up and ready to go build something.

DevOps, Because We’re Not In Kansas Anymore

DevOps is one of those things that are hard to wrangle down to a definition, and if you’ve read anything on the topic you probably didn’t walk away with a concrete understanding of what it is.  Most definitions seem incomplete if not even a little misleading. In particular, the main thing you hear is that DevOps is about increased collaboration between dev and ops teams. However many apps in the cloud are supported entirely by the dev team, with minimal or no ops involvement, but still very much have need for DevOps. In this context, DevOps is more about new tools, methods and practices of building and running apps in the cloud.

“I Would Spend 55 Minutes Defining the Problem and then Five Minutes Solving It”
Albert Einstein

Okay Einstein, let’s start there. The problem is we need to run and continually update our software in a scalable, reliable, and affordable manner. For this we look to the cloud. But running applications in the cloud is fundamentally different, and to do it effectively we need to adapt to a new way of doing things.

DevOps is the “stuff” that solves that problem.

So what is DevOps? The predominant, and tip of the iceberg answer is that DevOps is about delivering software faster, which in turn is about automation — having a robust continuous integration and testing process, automating the deployment process, and even automating the creation of infrastructure. It’s getting from “I just finished writing code” to getting it into an environment to test.

Before the cloud, there was often a sense of permanence and consummation in both the infrastructure and the application itself. The new reality is that both are ephemeral, subject to constant change and updating.

We do DevOps because we’re not moving into a house, we’re getting on a treadmill.

So sure, DevOps is about building a deployment pipeline, but that’s really just the beginning.  It’s the “day two” stuff, the operations, the scaling and monitoring, that I think is often overlooked as part of DevOps. To consider why we need new tools and methods, let’s reflect on what many of us were doing prior to the cloud. We had been running our applications on a fixed number of servers which we spent weeks to procure and configure, mostly by hand with little to no documentation or automation. We gave these servers cutesy names like they’re pets. Deployments often involved scheduled downtime. Servers were too often in a single geographic area.

Not surprisingly, a first endeavor to deploy in the cloud often involves as little adaptation as possible — we’re doing it a lot like we used to “back on Earth” by hand configuring servers, imaging them, and launching VM’s from those images. So we’re technically “in the cloud”, and there are some powerful benefits already, but we’re only halfway to solving our problem.

In the cloud we’re deploying on to VM’s or containers which may launch and terminate in as little as a few minutes. And they might be behind a load balancer and firewall that was just created a moment prior with an API call.  Scaling is elastic, measured, and operates automatically based on real-time conditions.

Launching an application into the cloud is a lot like launching a satellite into space.

Your app is running on instances on the east coast, then it’s on the west. Next time you look it might be in Asia. Getting information from it is like telemetry.  It’s a helpful mindset to consider servers in the cloud as being inaccessible remote locations because it reminds you to plan ahead what stats, data, and logs you want to get back from it and build an automated means of transmitting that data to a central place. You can’t wait until something goes wrong and then expect to ssh into the server and look around, it may well be terminated by then. In fact, just to remind you how short lived they are, we don’t even call them “servers” anymore, they’re “instances”.

Before cloud architectures, the assumption was the server was a “going concern”. It was going to be there and continue to operate unless something catastrophic happened. With cloud, this assumption is turned on its head. For reasons like auto-scaling, health check failure, or even getting your hourly price out bid in the spot market, your instance could be terminated in a moments notice. All of this means we now have to build our apps in a way that is tolerant to sudden shutdown and can resume by another node. We can’t even rely on short term storing of any state in memory or on disk.  Consider them literally as they are called, “compute instances”.

We have to have a strategy for health checking our applications. Sure we’ve had monitoring before, but now our infrastructure will automatically terminate any instances that fail to meet the health check requirements — and if you’re not careful, can terminate your entire cluster for something that might have been a momentary hiccup, leaving you completely down for 5 to 10 minutes until new VMs boot up.

These are among the many fascinating DevOps problems you will need to think through and solve as you devise your own way of deploying and scaling apps in the cloud.

All of a sudden you realize you’re building your own platform.

A more complete picture is that DevOps isn’t just automating deployments, it’s building or updating applications to 12 factor apps with scaling and ephemerality in mind, which implement your established health monitoring and logging standards, and building a pipeline and set of practices for deploying and operating those apps.  We don’t just get that out of the box with our IaaS provider.  And so I think a lot of the work which is currently done out of necessity under the umbrella of DevOps will increasingly be provided by platforms that sit a layer on top of IaaS.  Knowing how much work goes into building all of this on your own, I believe that PaaS is the future, for the same kinds of reasons why IaaS has value over building your own data center.

We need DevOps because we need the cloud, and the reality is that DevOps is just necessary to deploy and operate modern apps in the cloud, weather you are doing all of it yourself or leveraging a platform for some portion of these functions.

UX So Bad It Just Might Kill Someone: Audible’s New Clips Feature

Even with the recent rise in focus on user experience (UX) testing, I think the practice in general is frightfully under invested in. I don’t have unreasonably high expectations of products, but sometimes the UX is so bad that it makes me literally shout out loud in anger at the company who made it. Such was the case from my experience with the new Audible app this morning. I have listened to hundreds of books with Audible over the past several years. I’ve recommended Audible along with the practice of using your commute to learn to anyone who’ll listen. Lately I’ve been listening mostly to podcasts, so I didn’t notice the recent changes to the Audible app until this morning.

They’ve replaced “Bookmarks” with “Clips”. Sounds harmless. In fact, the clip feature can be useful, if say I’m listening to a book while lying back on a beach chair by the ocean where I can casually take the time to mark the beginning and end point of my clip and edit it to make sure I remove any unintended words. But like most people, I listen to audio books mainly in my car, and you can’t use this feature while driving without seriously risking your life. I can’t bother with marking the beginning and end of my clip you *$$@#!#@$#  $#!@%, I’m driving! I tried to use a clip like a bookmark but it resulted in making a large clip from the beginning of the book to the point where I wanted to bookmark.

It’s labeled in the app “Clips and Bookmarks”, seemingly implying think the plain old bookmark functionality is still there somewhere if I can just find it (it’s not).  It was dangerous enough using bookmarks while driving, but clipping is completely out of the question. It’s a shame because I really value my bookmarks.  But this feature is just too dangerous to use while driving.  I guess I will have to make sure the important points sink in the first time.  I just expect more from large companies like Audible (Amazon), in part because I know it’s likely that there’s a team of people who work there whose focus is UX.

I know, I’m not supposed to be using my phone while driving in the first place.  But I’m not sure how realistic that is as a hard rule.  This isn’t exactly texting.  I think certain apps like music, audio books, and GPS navigation, are meant to be used in a car and it’s unrealistic to expect zero user interaction while driving.  If these apps are made with the expectation of light use while driving, they should be no more difficult to use than operating the built in car radio or air conditioning.

2015: A year of gratitude journaling

In late 2014, I started keeping a gratitude journal.  I have an appointment on my calendar every weekday at 12:30 pm to remind me to do this.  I try to take a few minutes, usually while eating lunch, to journal things I’m grateful for that day.  I do this by emailing myself with the subject line “gratitude” and saving them into a folder.  In 2015 I had a total of 115 entries.  Here are the most frequently used words in those entries, in the form of a word cloud.
2015-gratitude-word-cloudI’ve read several books which encouraged the practice of gratitude journaling, but it was Oprah Winfrey’s 2014 book “What I Know For Sure”, which convinced me. I think her words say it best.


“Sometimes we get so focused on the difficulty of our climb that we lose sight of being grateful for simply having a mountain to climb. I know for sure that appreciating whatever shows up for you in life changes your whole world. You radiate and generate more goodness for yourself when you are aware of all you have and not focusing on your have nots. It isn’t easy, but it’s when you feel least thankful when you are most in need of what gratitude can give you — Perspective. Gratitude can transform any situation. It alters your vibration, moving you from negative energy to positive. It is the quickest, easiest, most powerful way to effect change in your life. This I know for sure. Here’s the gift of gratitude. In order to feel it, your ego has to take a back seat. What shows up in its place is greater compassion and understanding. Instead of being frustrated, you choose appreciation. And the more grateful you become, the more you have to be grateful for.”


If I could hope for any one thing for my daughters, it’s that they will someday choose to set aside time on a regular basis to acknowledge the things they’re grateful for. The hope of that literally brings a tear to my eye. I know that this practice has helped me appreciate life more in the present moment, and helps shift my thinking towards possibilities and opportunities in the future.

Why I Went Back To Being a Software Developer

When I was a kid, I was told that a midlife crisis was freaking out because you suddenly realized your life is half over, which results in buying a red sports car and running away with a new 20 year old girlfriend. While I’m not sure if 38 qualifies for midlife, I’ve certainly become acquainted with the concept, and I think a more accurate depiction of midlife crisis for some, is finding yourself nearly half way through your career and still not knowing exactly what you want to be “when you grow up”.

I started as a software developer and found initial success there but I really wanted to be on the business side.  So I founded my own software company which pivoted into an online retailer, then moved on to senior management roles in large established retail businesses running e-commerce and digital marketing. Now I’m going back to being an individual contributor in a software development role.  I feel like I owe at least myself a well articulated explanation for taking several steps back on the org chart, after investing nearly a decade on the business side of an e-commerce career path.

First, why I chose to move away from the business side of e-commerce.

Retail e-commerce is a largely undifferentiated space.

Prior to the Internet, retail stores had to compete mainly with other physically nearby stores. Many found success largely due to this limitation on the number of competitors.  Online the dynamic becomes, why should someone buy from you when they have hundreds of other options, most of them offering better pricing, better delivery times, more selection, and better customer service? It’s more important than ever to have a unique selling proposition.  You can’t just show up offering the same thing for sale that everyone else is selling.  You actually have to provide value (cheaper, faster, friendlier, more selection, unique selection, more helpful) to be successful.  That may sound obvious but nearly all online retailers bring nothing unique to the table.

Online retail is increasingly becoming a winner take all proposition, dominated by a handful of players.  The top fifty retailers in the U.S. control 75% of all e-commerce volume in the country, and the top four retailers (Amazon, Apple, Walmart, and Staples) control 40% of all e-commerce.  And as lines continue to blur between online and offline, this pattern will not be unique to retail.

The lesson for me is that if your business isn’t uniquely providing value, it is just as ephemeral as the EC2 instances it’s hosted on.

When you’re on the business side of e-commerce, the expectation from senior management is often to provide transformative results while only having the autonomy to make incremental improvements.  There’s this unspoken notion that you can just sprinkle e-commerce pixie dust on the website and extract your share of Internet riches.  You know, just find those perfect keywords, do that hipster magic in social media, and split test the color of your Add to Cart button until you find the optimal shade of green.  But all the best practices in the world are not going to move the needle if you’re not competitive on price, selection, or convenience.

Often there’s an over-emphasis on marketing to “get traffic” and “make that traffic convert”. Marketing’s role is to position ads to be in front of as many top quality purchase prospects as possible at the most effective cost of doing so, but it can’t get people to buy. That’s the job of the core business. And if the business you’re promoting isn’t better than its competitors in some meaningful way, you’re just not setup for success.

So why move back into tech?

Because the Internet has made the world so much smaller, the resulting winner take all pattern makes the company you work for really important, no matter what your role is.  At many of the best companies, engineering is valued above marketing, and is a better bet to finding yourself on a winning team.

I started to think about all the companies I most respect, and the pattern was they were all innovators in technology.  I realized that what I loved was working with the Internet, not retail, or selling the exact same thing everybody else is selling, or putting a SKU into a box and shipping it.

So does this mean I intend to be a geek for the rest of my days and have forever left the business side?  Not exactly.  As business and technology become more tightly coupled, I think this path will lead to different business opportunities – better ones.

Lastly, in regard to taking a step back from management, being a leader and being a manager are not necessarily the same thing.  Being a leader does not require a title, and sometimes you can have more influence as an individual contributor than you can as a manager.

Store Locator – The Most Underutilized Asset In Online Retail

Want a quick way to increase your online sales?  If you’re an omni-channel retailer, there’s probably something simple you can do right away and may see a 10% bump as a result.

Your store locator probably gets in the neighborhood of a quarter of your site traffic.  And nearly half of these store locator visitors bounce without exploring your site any further.  What a shame.

When a customer uses your store locator to look up a store, it’s a strong indicator of intent.

They’re not looking up your location for the fun of it, they’re doing it because they’ve already researched the product or service and are further down the funnel, ready to take action.  They’ve half made up their mind to buy from you and now they just need to get your address or your phone number so they can drive to your store.  Traditional retail folks know that once you’re physically in the store the conversion rate is very high, typically 30% or more, which baffles the minds of strictly online guys, and is a phenomenon worthy of its own research.  So that’s great but if you’re reading this you’re the e-commerce guy whose job is to grow the online business, at least to the point where it isn’t so damn pathetic as a percentage of the overall business’s revenue.

Here’s an easy score – put a call to action on your store locator pages which encourages customers to at least look at the online store before closing their browsers.  Do you have a sale running with a promo code on your store?  Convenience features like reserving items or pick up in store?  Highlight this from the store locator with a link to online shopping!  If you have any sort of service or consultation component to your business, by all means accept online appointment scheduling.  Customers browsing the store locator are primed and ready to convert.  Do some simple things to get more of them to convert online where you can track it (and before they change their minds).  These are typically some of the easiest changes you’ll make as we’re mainly talking about some creative and a hyperlink, and I’ll bet you can see results that outperform that last big feature you spent 4 months of development on.

So if your store locator does nothing more than give store location and contact information, for heavens sake do something to encourage them to maybe shop online before just closing their browsers.  You’ll be glad you did.  And while you’re at it, if you’re not making bi-weekly payments on your mortgage, sign up for that today too.  Seriously, you can take years off that thing and you get paid every two weeks anyway.  There, that’s two solid pieces of financial advice.  You’re welcome.