Introducing Riley & Bot: Jobs For Robots And Jobs For Me

I’m excited to announce the release of Jobs for Robots and Jobs for Me, an illustrated children’s book about artificial intelligence (AI) and the future of work.

I wrote this book because I wanted to introduce to children and parents the notion that AI will be capable of doing much of our work by the time the kids reading this book “grow up” and are ready to get a job, and because of my daughter Mackenzie’s interest in books on jobs (she wants to be a veterinarian). Many people seem to underestimate the impact and timeline that AI will have on jobs and our economy. It’s hard enough to believe that autonomous vehicles aren’t far away from displacing millions of transportation workers. It’s even more difficult to think that you might go through years of graduate school only to find that your area of discipline is performed better by AI. It’s a tough message. There is so much change coming so fast that it is hard for anyone to believe. We’re all children it seems when it comes to the understanding, appreciation, and fear of the power of AI.

I don’t try to paint a rosy picture just because it’s for kids. I think they are due a glimpse of how jobs will likely shift to robots and AI in the future. This book is real about the extent to which AI may largely displace jobs ranging from truck drivers to doctors, and even artistic and creative work will not be exempt.

While I playfully point out that there will be new jobs created which don’t exist yet, the world I hint at is one where AI will be better than us at many tasks and the owners of the AI are set to receive huge rewards while the rest of us struggle to shift into work still better performed by people. The story is a mixture of surprise, despair, excitement and hope, because it’s not a fairy tale, it’s a thoughtful depiction of the world I truly believe our children will find themselves in. We must reinvent ourselves as the landscape of work is reinvented before our eyes. Ultimately I hope for this book to be a conversation starter, and to raise awareness with parents and children so we can be best prepared for whatever is coming.

The book is available in softcover, hardcover (coming soon), and e-book at Amazon.

Wisdom from Will

Book Review: 'Will,” by Will Smith with Mark Manson - The New York Times

When you read someone else’s story you get the benefit of wisdom it took them their whole life to acquire.  It’s one of the reasons I love biographies.  I loved Will Smith’s new book, I couldn’t recommend it more.  It was entertaining and also contained wisdom that really resonated with me.  Below are all his words, I have no need to add to it.

The thing I’ve learned over the years about advice is that no-one can accurately predict the future, but we think we can.  So advice at its best is one person’s limited perspective of the infinite possibilities before you.  People’s advice is based on their fears, their experiences, their prejudices and at the end of the day their advice is just that, it’s theirs not yours.  When people give you advice, they’re basing it on what they can do, what they can perceive, and what they think you can do.  But the bottom line is, while yes it is true that we’re all subject to a series of universal laws, patterns, ties and currents, all of which are somewhat predictable, you are the first time you’ve ever happened.  You and now are a unique occurrence of which you are the most reliable measure of all the possibilities.
– Will Smith, Chapter Five: Hope

We punish ourselves for not knowing.  We always complain about what we could and should have done.  And how much of a mistake it was that we did that thing, that unforgivable thing.  We beat on ourselves for being so stupid, regretting our choices and lamenting the horrible decisions we make.  But here’s the reality – that’s what life is.  Living is the journey from not knowing to knowing, from not understanding to understanding, from confusion to clarity.  By universal design you are born into a perplexing situation, bewildered, and you have one job as a human – figure this shit out.  Life is learning.  Period.  Overcoming ignorance is the whole point of the journey.  You’re not supposed to know at the beginning.  The whole point of venturing into uncertainty is to bring light to the darkness of our ignorance.  I heard a great saying once.  Life is like school with one key difference.  In school you get the lesson and then you take the test.  But in life you get the test and it’s your job to take the lesson.  We’re all waiting until we have deep knowledge, wisdom, and a sense of certainty until we venture forth, but we’ve got it backwards.  Venturing forth is how we gain the knowledge.  
– Will Smith, Chapter Six: Ignorance

The universe is not logical.  It’s magical.  A major aspect of the pain and mental anguish we experience as humans is that our minds seek and often demand logic and order from an illogical universe.  Our minds desperately want shit to add up but the rules of logic do not apply to the laws of possibility.  The universe functions under the laws of magic.
– Will Smith, Chapter Ten: Alchemy

An alchemist is a spiritual chemist, a master of transmutation.  The great feat of an alchemist is that they can do the impossible.  They can turn lead into gold.  This concept erupted in my mind – the ability to take anything that life gives you and turn it into gold.  Gigi could take the last half glass of welch’s grape juice and mix it with the last swallow of dole pineapple juice, throw in some kool aid packets, dice up some lemon and the other half of the orange she was just eating and swirl it all together with a blast of canada dry ginger ale, freeze it, and hand you the best damn popsicle you’ve had in your life.  This was after you’d looked in the refrigerator five separate times and each time told her there was nothing in there.
– Will Smith, Chapter Ten: Alchemy

Change can be scary but it’s utterly unavoidable.  In fact impermanence is the only thing you can truly rely on.  If you are unwilling or unable to pivot and adapt to the incessant fluctuating tides of life, you will not enjoy being here.  Sometimes people try to play the cards that they wish they had instead of playing the hand they’ve been dealt.  The capacity to adjust and improvise is arguably the single most critical human ability.
– Will Smith, Chapter Eleven: Adaptation

“Do not get comfortable with your back on that canvas!”, he said.  “You fight how you train.”  You fight how you train was one of Darrell’s central axioms.  “You do everything how you do one thing”, he’d say.  Darrell didn’t want me to get comfortable with my back on the canvas in case I ever got knocked down.  He wanted lying down in a boxing ring to feel utterly foreign to me just in case I ever found myself lying down in a boxing ring.  His position was dreams are built on discipline, discipline is built on habits, habits are built on training, and training takes place in every single second of every situation of your life.  How you wash the dishes, how you drive a car, how you present a report at school or at work.  You either do your best all the time or you don’t.  If the behavior has not been trained and practiced then the switch will not be there when you need it.  “Training is for the purpose of habituating reactions to extreme circumstances”, Darrell said.  “When situations get hot you can’t rely on your thinking mind.  You must have habituated reflexive responses that kick in without the necessity of thought.  Never detrain your killer instincts.”
– Will Smith, Chapter Sixteen: Purpose (Darrell Foster who trained Will to portray Muhammad Ali)

Surrender had always been a negative word for me, it meant losing or failing or giving up.  But my burgeoning relationship with the ocean was exposing that my sense of control was actually an illusion.  Surrender transformed from a weakness word to an infinite power concept.  I had a bias toward action – thrusting, pushing, striving, struggling, doing.  And I began to realize that their opposites were equally as powerful – inaction, receptiveness, acceptance, nonresistance, being.  Stopping was equally as powerful as going.  Resting was equally as powerful as training.  Silence was equally as powerful as talking.  Letting go was equally as powerful as grasping.  Surrender to me no longer meant defeat.  It was now an equally powerful tool of manifestation.  Losing could be equal to winning in terms of my growth and development.  I began to understand a perplexing phrase that Gigi used to use – “let go and let God”.  That had always seemed wrong to me.  It felt like absolving yourself of your responsibilities, like something that people say when they are too lazy to do what’s necessary to build the life they want.  But all of a sudden it took on new and magical meaning.  There is an energy that is at work while you were asleep, the energy that fires the sun, that moves the ocean and that beats your heart.  You don’t have to do everything.  In fact most of the things that get done you didn’t have anything to do with them, and actually it’s a great thing that you were asleep because if you had been awake you probably would have messed it up.  Then a new wording of Gigi’s axiom came into mind, not just let go and let God, it’s let go and let God work.  The surfer and the ocean are a team.  The mountain and the climber are partners not adversaries.  The great river is going to do 99% of the work.  Your 1% is to study it, to understand it, to respect its power, and to creatively dance within its currents and its laws.  Act when the universe is open and rest when she’s closed.
– Will Smith, Chapter Twenty: Surrender

8 Quotes That Shaped My Worldview

silhouette photography of person

A worldview is a collection of attitudes, values, stories, and expectations. It represents our most fundamental beliefs and assumptions about the world around us. “As such, worldviews play a central and defining role in our lives. They shape what we believe and what we’re willing to believe, how we interpret our experiences, how we behave in response to those experiences, and how we relate to others”, says James Anderson of Ligonier Ministries.

These thoughts have greatly shaped how I see the world.

“The fool doth think he is wise, but the wise man knows himself to be a fool.” (Shakespeare)

It’s easy to get caught in the trap of thinking we know what is best for ourselves, for others, and the world. The best we can do is have some humility the next time we catch ourselves thinking we know the way. It’s okay not to have all the answers – better than okay. Acknowledging this may be our only chance at wisdom.

“Security is mostly a superstition. It does not exist in nature, nor do the children of men as a whole experience it. Avoiding danger is no safer in the long run than outright exposure.” (Hellen Keller)

There is no “safe”. Everything you have could be gone in a minute for reasons beyond your control, whether you play it safe or not. If you knew that savings account you’ve been contributing to for so many years would be wiped out when you needed it, would you regret not having taken some risks along the way? That is not to say to be reckless, but being risk averse doesn’t pay the life long dividends you think it does, and likely steals the best version of yourself away.

“You shouldn’t hesitate to modify your perceptions to whatever makes you happy, because you’re probably wrong about the underlying nature of reality anyway.” (Scott Adams)

Part of why we’re so hard on ourselves is we think we understand the world. We think we know its rules and limitations, and our own potential or lack thereof. But what if we’re wrong about all of that? What if we’re just not capable of understanding the true nature of our world or how others perceive us?

Scott continues, “If I had to bet my life, I’d say humans are more like my dog trying to use psychic powers on me to play fetch than we are like enlightened creatures that understand their environment at a deep level.” This worldview helps me not to take things so seriously. I believe there is more magic, wonder, and possibilities both within the universe and within ourselves than we are capable of seeing or understanding.

“Believe those who seek the truth, doubt those who find it.” (Andre Gide)

Confirmation bias is an inescapable part of humanity. We don’t seek the truth so much as we seek out things that support the conclusions we’ve already made. It’s sad that it takes so little to form an initial opinion and that all information seen after developing this opinion is seen as either “right” and in support of our early conclusion, or “wrong”. This is true even in the field of data science where we can masterfully manipulate the data into telling whatever story we want. And just being aware of confirmation bias doesn’t make us immune to it. I hope to be as open-minded to concede to the possibility that everything I know might be wrong, but I’m probably too entrenched in my views to truly be that objective.

“I find it helpful to remind myself that every human is a mess on the inside. It’s easy to assume the good-looking and well-spoken person in front of you has it all together and is therefore your superior. The reality is that everyone is a basket case on the inside. Some people just hide it better. Find me a normal person and I’ll show you someone you don’t know that well. It helps to remind yourself that your own flaws aren’t that bad compared with everyone else’s.” (Scott Adams)

Enough said.

“Be, Do, Have.” (Robert Kiyosaki)

Before you can DO what is necessary to HAVE the life you want, you must BECOME the person who would do those things. The order is very important. I used to think if you do certain things you can have what you want and that will make you into the person you want to be. But having things doesn’t make you into anything, nor would you likely do enough to have what you want without first becoming the person you need to be.

The “be” is a flywheel. If you’re a _____ kind of person, you can’t help but do _____, and if you do that you can’t help but have _____. It’s automatic. You can’t help but do what you do because of who you are. This can work to your benefit or against it, depending on what your “be” truly is.

“Life will give you whatever experience is most helpful for the evolution of your consciousness. How do you know this is the experience you need? Because this is the experience you are having at the moment.” (Eckhart Tolle)

Have a difficult person in your life? I believe this is a gift, handcrafted and placed exactly into your life so that you can learn from it. Sometimes it can take years to understand the lessons from the difficult things in your life, maybe a lifetime, and life will keep serving you the same kind of lesson until you get from it what you need.

“For even a few moments at a time, being aware of what is arising, gives us the possibility to make a choice.” (Joseph Goldstein)

I’m not so sure we actually have free will, or maybe that we are not really conscious the vast majority of the time. So rarely are we present enough to make a choice we could truly call our own. The best we can do is focus on the present moment, notice our own thoughts as they arise and maybe instead of just reacting and following our program, choose if those desires are really from our innermost soul.

See also Why You Probably Don’t Have Free Will.

Things That Trigger Me (Well, Tech Related)

After I left my last company, one of my teammates shared with me a list of things she had been keeping about me in the Notes app in her phone called “Things that trigger Doug”. It must say something about me that someone would keep such a list. Anyway it made me laugh! You’ll only appreciate this if you work in tech. I give it to you unedited:

  • Writing shitty code the first time with the excuse of “trying to get it to work” / hard coding shit to make it work
  • White IDE theme
  • Stealing his phone
  • Waiting in a line for 2 hours
  • When you don’t believe in containers
  • Windows 7 in 2019
  • Using private chats instead of public Slack channels
  • Agile words
  • People who wait to write their Dockerfile until the last minute /don’t dev in a docker container
  • People cheating in the step challenge when he runs 14–16 miles a day
  • Designated CAB meetings /calling releases “emergency” or “off cycle”
  • Notepad++
  • QA tester boxes / manual testing
  • Ad blocker. Support the creators!
  • Naming things poorly
  • End to end tests
  • People calling stuff a blocker when they aren’t blocked by shit
  • Iframes
  • Wasting money in the name of frugality
  • Putty
  • Putting “Personal Interests” on your resume

10 Rules For Building Better Internet Applications

blog_headerIf you’re working on an application that’s been around for a while, has lots of usage and several different teams working on it, there’s a good chance it’s a dumpster fire. It happens to the best of us. Building software with real customer adoption is messy business. While there is no one size fits all, these are my opinions on building more reliable, scalable, and maintainable Internet applications.

1. Go All In On One Cloud

Pick the leading cloud provider that makes the most sense for your team and go all in. Don’t miss half the benefit of the cloud by limiting yourself to the lowest common denominator between cloud providers, which is basically just VMs. This is folly anyway as it is nearly impossible to build an application natively in the cloud without tentacles into proprietary resources from that provider. Trying to be on the light end of the lock in spectrum is not worth the trade off you’d be making in lost productivity and reliability.

2. Forget About Servers

Use containers and/or serverless for 100% of your dev, stage, and prod environments. There is almost no reason to use VMs directly for any new development. While containers and serverless have their own set of challenges they are an order of magnitude better than using configuration management (Ansible, Chef, Puppet) and are your only hope for getting your dev team to participate in DevOps, which is necessary to scale development of a cloud native application.

3. Commit Wholly To Infrastructure As Code

Don’t take shortcuts that question your ability to stand up the application from source in the future. Make defining your infrastructure as code a requirement, not a nice to have. This applies to your infrastructure primitives (e.g. building blocks directly in your AWS account) as well as any third party tools or services you may want to integrate with. If it can’t be brought to life entirely from source code, you should seriously consider an alternative that can. Your application code needs a specifically configured environment to run in. Setting up that environment by hand is like writing a program, deploying it to production, then deleting the source code. Let’s not even talk about that install guide document you pretend is accurate and up-to-date.

Your first line of code should be your Dockerfile, or maybe your serverless yaml or your Terraform. The point is, start with your infrastructure code, at least enough to define your immediate development environment, which becomes the blueprint for production. Store your infrastructure code with your app code, lock step with the application in the same git repo with each microservice. You should be able to clone a git repo from any computer, build the Docker image and immediately start working on it, no matter what language it’s written in or dependencies it has. “It works on my machine” is the joke of the last generation of software development. Think of your machine as a text editor and a Docker engine.

4. Embrace The Ephemeral

I think it’s helpful to think of your code running on a “server” that didn’t exist 5 minutes ago and won’t exist 5 minutes from now (or maybe seconds, or milliseconds). If you haven’t noticed, in the cloud they aren’t even called servers anymore, they’re instances, hinting at their proper use lifespan.

Never write out any state to local disk volumes. Pretend your environment doesn’t even have a local disk. Expect instances of your program to terminate abruptly due to auto-scaling or chaos experiments. Your operations need to be idempotent, and anticipate retries. You need good logging and usage metrics, which your application must be deliberately broadcasting. Throw away your ssh keys and keep the telemetry mindset of your application running on an inaccessible remote location.

This forces the issue of immutable infrastructure. It makes no sense to modify or “patch” something that has a lifespan of milliseconds to minutes. Change is always by terminating the existing resource and replacing it with a new one with new settings.

5. Use Managed Services Wherever Possible

Building and operating clusters of anything isn’t providing business value or helping you to quickly iterate. Wherever possible, skip all that undifferentiated heavy lifting and use a managed service.

Installing a database server for example, on a single server for development isn’t so hard, but doing it in production in a way that is highly available is another story entirely. You have to setup an auto-scaling configuration where nodes can launch and rejoin the cluster, setup and test failover and restore for your HA configuration, and setup and test all the associated monitoring and alarms. It’s easy to get something wrong here and since you’re usually in the data layer, these are costly mistakes. And since our dev and stage environments need to match our production environment exactly (apart from instance size), the easy single server installation isn’t acceptable in any environment. Maybe this would be manageable if we were doing this once for one huge database server, but in a microservices world each with their own isolated data layer, it’s just not practical to self manage with dozens of services. Recalling the earlier rule about bringing everything to life entirely from infrastructure code, usually the only way to accomplish this without a herculean effort is to use a managed service. If you’re insisting on installing everything on EC2 yourself you are missing half the benefit of the cloud.

6. Testing Should Make You Go Faster Not Slower

Testing should give you rails so you can speed, not gates to force you to come to a stop. Your tests (all of them – unit, integration, load, front end) should be a completely automated part of your build and deploy pipeline, and like your infrastructure code, should reside in the same git repo as the component they are testing. The same integration tests should run against all environments, especially production, which is the most important environment to test because it is the only one your customers actually see. This means you need to use test accounts in your real databases, your testing can’t rely on starting from a fresh database each run, which isn’t feasible in production, and demands exclusive use of an environment for testing, which obviously doesn’t scale. This requires a close partnership with development, it simply won’t happen if you treat QA as a siloed activity.

Testing should share the microservices mindset – when we replace a pipe in the basement, we don’t always need to reinspect the whole house including the windows in the attic. Testing is critical, but we don’t want testing at any cost. Don’t let outdated testing practices be the weak link in your agile process.

7. Always Be Deploying

Because I think this one is so important, I’m going to borrow the sentiment from the infamous Glengarry Glen Ross, “Because only one thing counts in this life. Get [the code deployed to production]! You hear me you ***** *****! A, B, [D]. A – always, B – be, [D] – [deploying]. Always be [deploying]. Always be [deploying]!”

The more time that passes between deployments, the higher your integration risk, and the greater chance you are blocking another service which you are a dependency for. This is exactly why we do CI/CD in development. While I’m not saying you need to do continuous deployment in production, the same risks accumulate as time goes on so you want to keep the change delta as low as possible. Plan on deploying every sprint. Most deployments are back-end services with minimal to no customer facing aspects, so this doesn’t necessarily mean visible UI changes all the time.

Each microservice should be independently testable and deployable. When working on a feature that spans multiple microservices, deploy each one as soon as it is ready, don’t squirrel them away to deploy all at once in some blaze of glory.

So put that coffee down, coffee is for deployers!

8. Write Your README Before Writing Any Code

As the saying goes, “If you don’t know where you’re going, any road will take you there.” I love the Amazonian practice of starting a new project or feature by first writing a press release announcing it. I think that causes you to think bigger and to go into the work with a more clear picture of the most important parts of what you’re building. Perhaps the microservice equivalent of that is writing your README first. Give a couple sentences about what the thing does. Show usage examples demonstrating the main interfaces. Show examples of how to build, test, and deploy it. Do this first, in the README in your git repo, not last in some external document management system.

And for the love of God, before any code is written, think about how you’re naming things. Nothing shows lack of thought about how your service fits into the bigger picture of your ecosystem of services like giving it a vague or misleading name. Renaming things seldom happens, or is really expensive to do later.

9. Open Source From The Beginning

Even if you don’t want third party contributions, I think open sourcing your project reinforces a few helpful perspectives. First you are forced to handle secrets appropriately, which makes it a lot easier to get a short term consultant involved, because you can actually share your code base without worrying about changing the secrets later (let’s be honest, doesn’t always happen). Open sourcing your project may also inspire confidence from your customers about the longevity of your platform and inspire the best from your developers who will hold themselves to a higher standard knowing their work will be on display.

More importantly, it serves as a constant reminder that while it takes valuable skill to produce the source code, it has no intrinsic value and can be easily duplicated. I know this is a difficult pill to swallow when you’re spending boatloads of money on development but it’s the truth. Your value is in your data set, relationships, reputation, and execution.

At least look to open source components of your application which may be general use tools, or with a small amount of additional thought could have a general use interface rather than being specific to your application.

10. Iterate on Customer Experience

Andy Jassy ended his 2018 AWS re:Invent keynote with some remarks which encapsulates what this is all about.

“For any of us who are trying to build long standing sustainable businesses, which is hard, the most important thing by far is to listen really carefully to what your customers want from you and how they want the experience to improve, and then to be able to experiment and innovate at a really rapid clip to keep improving the customer experience. That is the only way that any of us will be able to survive over a long period of time. That’s what you should care about, giving your builders the most capable platform and allow them to keep iterating on that customer experience.”

While of course he’s suggesting that AWS specifically be the foundation of that platform, the sentiment rings true, and that’s what all of this is ultimately about, investing our resources iterating on customer experience, and not get bogged down with boring IT stuff.

Super Powers: AWS re:Invent 2016

Amazon knows how to throw a party. They hired the best DJ in the world, literally – Martin Garrix. The re:Invent Party was an experience I’ll never forget. Walking in, you are handed an activity map, showing the locations of the digital lounge, beer garden, dance floor, and play zone including a laser dome, ball pit, bubble soccer, climbing wall, and a labyrinth. But once I got to the DJ area, I was mesmerized.  I stood in the crowd and soaked in the music, the lights, and the special effects. It was truly a performance. Even though I traveled alone and didn’t know anyone there, I felt connected, surrounded by thousands of other like minded people celebrating at this amazing event.  AWS CEO Andy Jassy captured the sentiment best saying simply “it is awesome to be here.”

The conference was sold out at 32,000 attendees and occupied most of the Venetian and the Mirage resorts.  Everywhere you went it seemed like Las Vegas was invaded by swarms of cloud enthusiasts all wearing the black and blue AWS hoodie given to us on our first day.  Fredric Paul from New Relic put it well saying “The whole event has the palpable buzz of a celebration of a successful, growing industry that feels like it’s on the right side of history.”

The theme of Andy Jassy’s keynote was that AWS gives you super powers, and I think it’s true. You have access to essentially infinite resources. As a single developer you can build something over the weekend that you can deploy globally, that can auto scale to handle any amount of volume, store any amount of data, and initially cost next to nothing.  Want to cut your teeth on a 128 CPU server with 2TB of memory?  Spin it up right now, it’s yours for $13.34 per hour.  Where else would most of us ever even have access to a machine like that?  How about a fleet of them on the spot market for up to 90% off?  Need to accumulate 100GB of storage per day?  No problem, write it out to S3 for just $0.03 per GB per month, and know you’ll never run out of storage space.  Werner Vogels, CTO of AWS put it this way – “One of the things the cloud has done is democratize access to technology power.” Just knowing that I can affordably access any resources I need challenges me to tackle something of scale.

Better yet, as Jeff Lawson, CEO of Twilio, astutely pointed out, “you don’t get points for using servers, you only get points for serving users”.  With serverless architecture, you can put together entire solutions on inherently scalable building blocks that can serve millions of users, but only cost as much as a Starbucks coffee until it starts to get some traction. The cloud really has given us super powers.

Jassy wrapped up his keynote with a few words on the next 10 years. Because AWS has grown so substantially in its first 10 years, would the the next 10 will have less invention? “I believe the next 10 years will have markedly more innovation,” Jassy said.  Apparently, the cloud is just getting started.  Instead of debating wheather companies would use the cloud or not, Jassy said “They will spend all their time using the cloud and its massive capabilities to build new business that we never imagined before.” He continued saying “This is a world that I want to live in. And this is a world that I think is very exciting to everyone in this room.” I had a blast at re:Invent, and left pumped up and ready to go build something.

DevOps, Because We’re Not In Kansas Anymore

DevOps is one of those things that are hard to wrangle down to a definition, and if you’ve read anything on the topic you probably didn’t walk away with a concrete understanding of what it is.  Most definitions seem incomplete if not even a little misleading. In particular, the main thing you hear is that DevOps is about increased collaboration between dev and ops teams. However many apps in the cloud are supported entirely by the dev team, with minimal or no ops involvement, but still very much have need for DevOps. In this context, DevOps is more about new tools, methods and practices of building and running apps in the cloud.

“I Would Spend 55 Minutes Defining the Problem and then Five Minutes Solving It”
Albert Einstein

Okay Einstein, let’s start there. The problem is we need to run and continually update our software in a scalable, reliable, and affordable manner. For this we look to the cloud. But running applications in the cloud is fundamentally different, and to do it effectively we need to adapt to a new way of doing things.

DevOps is the “stuff” that solves that problem.

So what is DevOps? The predominant, and tip of the iceberg answer is that DevOps is about delivering software faster, which in turn is about automation — having a robust continuous integration and testing process, automating the deployment process, and even automating the creation of infrastructure. It’s getting from “I just finished writing code” to getting it into an environment to test.

Before the cloud, there was often a sense of permanence and consummation in both the infrastructure and the application itself. The new reality is that both are ephemeral, subject to constant change and updating.

We do DevOps because we’re not moving into a house, we’re getting on a treadmill.

So sure, DevOps is about building a deployment pipeline, but that’s really just the beginning.  It’s the “day two” stuff, the operations, the scaling and monitoring, that I think is often overlooked as part of DevOps. To consider why we need new tools and methods, let’s reflect on what many of us were doing prior to the cloud. We had been running our applications on a fixed number of servers which we spent weeks to procure and configure, mostly by hand with little to no documentation or automation. We gave these servers cutesy names like they’re pets. Deployments often involved scheduled downtime. Servers were too often in a single geographic area.

Not surprisingly, a first endeavor to deploy in the cloud often involves as little adaptation as possible — we’re doing it a lot like we used to “back on Earth” by hand configuring servers, imaging them, and launching VM’s from those images. So we’re technically “in the cloud”, and there are some powerful benefits already, but we’re only halfway to solving our problem.

In the cloud we’re deploying on to VM’s or containers which may launch and terminate in as little as a few minutes. And they might be behind a load balancer and firewall that was just created a moment prior with an API call.  Scaling is elastic, measured, and operates automatically based on real-time conditions.

Launching an application into the cloud is a lot like launching a satellite into space.

Your app is running on instances on the east coast, then it’s on the west. Next time you look it might be in Asia. Getting information from it is like telemetry.  It’s a helpful mindset to consider servers in the cloud as being inaccessible remote locations because it reminds you to plan ahead what stats, data, and logs you want to get back from it and build an automated means of transmitting that data to a central place. You can’t wait until something goes wrong and then expect to ssh into the server and look around, it may well be terminated by then. In fact, just to remind you how short lived they are, we don’t even call them “servers” anymore, they’re “instances”.

Before cloud architectures, the assumption was the server was a “going concern”. It was going to be there and continue to operate unless something catastrophic happened. With cloud, this assumption is turned on its head. For reasons like auto-scaling, health check failure, or even getting your hourly price out bid in the spot market, your instance could be terminated in a moments notice. All of this means we now have to build our apps in a way that is tolerant to sudden shutdown and can resume by another node. We can’t even rely on short term storing of any state in memory or on disk.  Consider them literally as they are called, “compute instances”.

We have to have a strategy for health checking our applications. Sure we’ve had monitoring before, but now our infrastructure will automatically terminate any instances that fail to meet the health check requirements — and if you’re not careful, can terminate your entire cluster for something that might have been a momentary hiccup, leaving you completely down for 5 to 10 minutes until new VMs boot up.

These are among the many fascinating DevOps problems you will need to think through and solve as you devise your own way of deploying and scaling apps in the cloud.

All of a sudden you realize you’re building your own platform.

A more complete picture is that DevOps isn’t just automating deployments, it’s building or updating applications to 12 factor apps with scaling and ephemerality in mind, which implement your established health monitoring and logging standards, and building a pipeline and set of practices for deploying and operating those apps.  We don’t just get that out of the box with our IaaS provider.  And so I think a lot of the work which is currently done out of necessity under the umbrella of DevOps will increasingly be provided by platforms that sit a layer on top of IaaS.  Knowing how much work goes into building all of this on your own, I believe that PaaS is the future, for the same kinds of reasons why IaaS has value over building your own data center.

We need DevOps because we need the cloud, and the reality is that DevOps is just necessary to deploy and operate modern apps in the cloud, weather you are doing all of it yourself or leveraging a platform for some portion of these functions.

UX So Bad It Just Might Kill Someone: Audible’s New Clips Feature

Even with the recent rise in focus on user experience (UX) testing, I think the practice in general is frightfully under invested in. I don’t have unreasonably high expectations of products, but sometimes the UX is so bad that it makes me literally shout out loud in anger at the company who made it. Such was the case from my experience with the new Audible app this morning. I have listened to hundreds of books with Audible over the past several years. I’ve recommended Audible along with the practice of using your commute to learn to anyone who’ll listen. Lately I’ve been listening mostly to podcasts, so I didn’t notice the recent changes to the Audible app until this morning.

They’ve replaced “Bookmarks” with “Clips”. Sounds harmless. In fact, the clip feature can be useful, if say I’m listening to a book while lying back on a beach chair by the ocean where I can casually take the time to mark the beginning and end point of my clip and edit it to make sure I remove any unintended words. But like most people, I listen to audio books mainly in my car, and you can’t use this feature while driving without seriously risking your life. I can’t bother with marking the beginning and end of my clip you *$$@#!#@$#  $#!@%, I’m driving! I tried to use a clip like a bookmark but it resulted in making a large clip from the beginning of the book to the point where I wanted to bookmark.

It’s labeled in the app “Clips and Bookmarks”, seemingly implying think the plain old bookmark functionality is still there somewhere if I can just find it (it’s not).  It was dangerous enough using bookmarks while driving, but clipping is completely out of the question. It’s a shame because I really value my bookmarks.  But this feature is just too dangerous to use while driving.  I guess I will have to make sure the important points sink in the first time.  I just expect more from large companies like Audible (Amazon), in part because I know it’s likely that there’s a team of people who work there whose focus is UX.

I know, I’m not supposed to be using my phone while driving in the first place.  But I’m not sure how realistic that is as a hard rule.  This isn’t exactly texting.  I think certain apps like music, audio books, and GPS navigation, are meant to be used in a car and it’s unrealistic to expect zero user interaction while driving.  If these apps are made with the expectation of light use while driving, they should be no more difficult to use than operating the built in car radio or air conditioning.

2015: A year of gratitude journaling

In late 2014, I started keeping a gratitude journal.  I have an appointment on my calendar every weekday at 12:30 pm to remind me to do this.  I try to take a few minutes, usually while eating lunch, to journal things I’m grateful for that day.  I do this by emailing myself with the subject line “gratitude” and saving them into a folder.  In 2015 I had a total of 115 entries.  Here are the most frequently used words in those entries, in the form of a word cloud.
2015-gratitude-word-cloudI’ve read several books which encouraged the practice of gratitude journaling, but it was Oprah Winfrey’s 2014 book “What I Know For Sure”, which convinced me. I think her words say it best.

“Sometimes we get so focused on the difficulty of our climb that we lose sight of being grateful for simply having a mountain to climb. I know for sure that appreciating whatever shows up for you in life changes your whole world. You radiate and generate more goodness for yourself when you are aware of all you have and not focusing on your have nots. It isn’t easy, but it’s when you feel least thankful when you are most in need of what gratitude can give you — Perspective. Gratitude can transform any situation. It alters your vibration, moving you from negative energy to positive. It is the quickest, easiest, most powerful way to effect change in your life. This I know for sure. Here’s the gift of gratitude. In order to feel it, your ego has to take a back seat. What shows up in its place is greater compassion and understanding. Instead of being frustrated, you choose appreciation. And the more grateful you become, the more you have to be grateful for.”

If I could hope for any one thing for my daughters, it’s that they will someday choose to set aside time on a regular basis to acknowledge the things they’re grateful for. The hope of that literally brings a tear to my eye. I know that this practice has helped me appreciate life more in the present moment, and helps shift my thinking towards possibilities and opportunities in the future.

Why I Went Back To Being a Software Developer

When I was a kid, I was told that a midlife crisis was freaking out because you suddenly realized your life is half over, which results in buying a red sports car and running away with a new 20 year old girlfriend. While I’m not sure if 38 qualifies for midlife, I’ve certainly become acquainted with the concept, and I think a more accurate depiction of midlife crisis for some, is finding yourself nearly half way through your career and still not knowing exactly what you want to be “when you grow up”.

I started as a software developer and found initial success there but I really wanted to be on the business side.  So I founded my own software company which pivoted into an online retailer, then moved on to senior management roles in large established retail businesses running e-commerce and digital marketing. Now I’m going back to being an individual contributor in a software development role.  I feel like I owe at least myself a well articulated explanation for taking several steps back on the org chart, after investing nearly a decade on the business side of an e-commerce career path.

First, why I chose to move away from the business side of e-commerce.

Retail e-commerce is a largely undifferentiated space.

Prior to the Internet, retail stores had to compete mainly with other physically nearby stores. Many found success largely due to this limitation on the number of competitors.  Online the dynamic becomes, why should someone buy from you when they have hundreds of other options, most of them offering better pricing, better delivery times, more selection, and better customer service? It’s more important than ever to have a unique selling proposition.  You can’t just show up offering the same thing for sale that everyone else is selling.  You actually have to provide value (cheaper, faster, friendlier, more selection, unique selection, more helpful) to be successful.  That may sound obvious but nearly all online retailers bring nothing unique to the table.

Online retail is increasingly becoming a winner take all proposition, dominated by a handful of players.  The top fifty retailers in the U.S. control 75% of all e-commerce volume in the country, and the top four retailers (Amazon, Apple, Walmart, and Staples) control 40% of all e-commerce.  And as lines continue to blur between online and offline, this pattern will not be unique to retail.

The lesson for me is that if your business isn’t uniquely providing value, it is just as ephemeral as the EC2 instances it’s hosted on.

When you’re on the business side of e-commerce, the expectation from senior management is often to provide transformative results while only having the autonomy to make incremental improvements.  There’s this unspoken notion that you can just sprinkle e-commerce pixie dust on the website and extract your share of Internet riches.  You know, just find those perfect keywords, do that hipster magic in social media, and split test the color of your Add to Cart button until you find the optimal shade of green.  But all the best practices in the world are not going to move the needle if you’re not competitive on price, selection, or convenience.

Often there’s an over-emphasis on marketing to “get traffic” and “make that traffic convert”. Marketing’s role is to position ads to be in front of as many top quality purchase prospects as possible at the most effective cost of doing so, but it can’t get people to buy. That’s the job of the core business. And if the business you’re promoting isn’t better than its competitors in some meaningful way, you’re just not setup for success.

So why move back into tech?

Because the Internet has made the world so much smaller, the resulting winner take all pattern makes the company you work for really important, no matter what your role is.  At many of the best companies, engineering is valued above marketing, and is a better bet to finding yourself on a winning team.

I started to think about all the companies I most respect, and the pattern was they were all innovators in technology.  I realized that what I loved was working with the Internet, not retail, or selling the exact same thing everybody else is selling, or putting a SKU into a box and shipping it.

So does this mean I intend to be a geek for the rest of my days and have forever left the business side?  Not exactly.  As business and technology become more tightly coupled, I think this path will lead to different business opportunities – better ones.

Lastly, in regard to taking a step back from management, being a leader and being a manager are not necessarily the same thing.  Being a leader does not require a title, and sometimes you can have more influence as an individual contributor than you can as a manager.