What is a Layer 2 Cloud Provider?

The rise of “cloud infrastructure” has presented a dilemma for developers: what is the appropriate level of complexity for a cloud provider to handle? In the last decade, the options for cloud engineering services have expanded exponentially, leading to a vast array of product offerings. While the variety and complexity offered by cloud providers such as AWS presents advantages to developers with economies of scale and scope, these same factors may be unnecessary or burdensome to other firms that seek a more focused and streamlined experience. 

Heroku is Layer 2 Cloud provider which provides a “Platform-as-a-Service.” Heroku was founded in 2007 and has evolved alongside the large cloud providers such as AWS. While Heroku is in some ways a competitor to large cloud providers for clients who wish to deploy web applications, it also builds on AWS cloud infrastructure and acts as a “middleman” between the cloud provider and the user. Heroku’s focus is on 12-Factor Web Apps, which is a set of guiding principles for building web applications that Heroku engineer Mark Turner said “really does define what describes an application suitable for Heroku.”

 

Given the scope and competitive power of cloud providers, Heroku’s longevity and continued use indicate its users find lasting value that is worth the additional marginal costs. We have conducted several interviews with employees at Heroku in the past several years, and each has had a unique take on Heroku’s value proposition, but a few major themes stood out.

Full disclosure: Heroku is a sponsor of Software Engineering Daily.

In the late 2000s, the rise of cloud computing service offerings such as Amazon’s EC2 heralded a shift in how software was deployed and managed over the web. Formerly, so-called “bare metal” servers were built in on-premise data centers that would act as the host for a company’s web software. Cloud providers such as Amazon Web Services offered “virtualization” services, whereby server hardware was partitioned by a hypervisor into virtual machines, which could be allocated to client applications for hosting.

 

Virtualization abstracted away the underlying hardware management; subsequently, cloud providers built infrastructure services to interact with and manage VMs. This gave rise to Infrastructure-as-a-Service offerings, which “provide high-level APIs used to dereference various low-level details of underlying network infrastructure like physical computing resources, location, data partitioning, scaling, security, backup, etc.” Cloud providers that focus on the management of hardware up through the IaaS level are called “Layer 1 Cloud Providers.” Layer 1 cloud providers require significant scale and scope to operate effectively due to the complex operational challenges of managing hardware and server operations at a granular level.

 

Today, Amazon Web Services is the dominant Layer 1 cloud provider. AWS accounts for nearly half of the market for IaaS providers. The second-largest Layer 1 provider, Microsoft Azure, trails at a distant 15.5%. AWS currently offers 212 services on its platform, and offers an array of certification courses training developers and “solutions architects” on its various products.

 

Above is a diagram of a simple application architecture deployed with AWS services. As applications grow in scale, the number and complexity of the services used can increase dramatically. The advantages of precise configuration and complex architectures are significant for companies with large development and operations teams matched with broad and active customer bases. However, growth in the scope of services creates challenges of its own. As Mark Turner put it:

“Everything that companies like AWS offer as the wide disparate services that they provide means that you really get a giant kit of solutions to problems that you don’t even know you have sometimes.”

Heroku is a platform-as-a-service (PaaS) provider that focuses on providing a streamlined developer experience for deploying and managing web applications on the cloud. Heroku is recognized as the first and most prominent Layer 2 cloud provider, and currently has 16,171 websites deployed via its service. Heroku is built on top of AWS EC2 instances, and seeks to act as a layer of abstraction between users and underlying IaaS providers. Heroku infrastructure engineer Jon Daniel told us:

“We are essentially taking on the role of your operations team, we’re doing a lot of preventative maintenance. We’re doing a lot of just things that you would never think about behind-the-scenes, and that’s sort of what we bake into our costs.”

Streamlining the developer experience does not translate to simplifying the underlying infrastructure; in fact, the task of adapting layer 1 technologies to a layer 2 interface is a difficult engineering task. Heroku manages several of the underlying infrastructural tasks that a firm may face when managing a web application. For example, Heroku manages application scaling, provides access to autoscaling, and runs a metrics pipeline on Kafka and Cassandra to provide “health checks” essential to an autoscaling process. Heroku engineer Andy Appleton noted that “the entire Heroku product is a developer product,” and that the focus was on creating the best possible UX to add value on top of the Layer 1 AWS cloud framework. Heroku also offers several services such as managed Postgres databases, Redis, version control, and continuous deployment. All of these services and more may be easily integrated with a minimum of time spent managing configuration. Mark Turner described some of the operational needs of Heroku customers:

“We might have customers that have workloads, where each single process uses 14 gigs of memory. At the same time, they might have something that where each process uses 2 megabytes and they want each of those things to schedule and boot up instantaneously. That’s the orchestration layer problems we deal with that makes it hard.

Then it’s the isolation and security boundaries between all of that stuff that also makes it hard, and auditing and patching and maintaining those boundaries is the hard part. Then you factor in layering on those workflows that power that Heroku experience, that those containers encapsulate down at the bottom of it; it all adds up into I think just a hard system to build.”

When a cloud platform expands, it can do so either “horizontally” or “vertically.” Horizontal scaling is the expansion of scope. For example, Amazon adding new available functions to its cloud platform represents scope expansion. The horizontal expansion allows a cloud provider to tackle new problems, or to solve old problems in more efficient ways. The adoption of Kubernetes for container orchestration represented horizontal expansion for Layer 1 cloud providers- including Google’s GKE and AWS’s EKS. Heroku has also undergone horizontal expansion, adding in services such as Kafka to its suite of products. We spoke with Tom Crayford, an engineer at Heroku, at length about adapting Kafka to a managed, streamlined Heroku experience. Despite the scope expansion, Heroku makes efforts to focus its expanded offerings on products that work in mostly the same way in order to minimize the work necessary to add or switch services. From Jon Daniel, talking about Heroku’s managed Postgres offerings:

“The nice thing about that is being able to have a fairly standard, almost single-tenant configuration set up on that instance, and the configuration changes themselves as to like what version of Postgres or what plan type is really based on mounting. So we know that everything is going to act very similarly from one to the other and there’s not a lot of one-off configuration happening in there.”

On the other hand, vertical expansion is “creep up the developer experience stack,” which is when a cloud provider deepens its service offerings by creating new abstractions on top of the base layer. It’s worth noting that the cloud itself, as we think about it in terms of modern software engineering, is a layer of abstraction on top of server hardware. From Mark Tuner:

“Where we expend our energy in leveling up the platform’s capabilities is really important. We pay a lot of attention to how we spend that energy. It’s really important for us to make calls that aren’t good for us, because there’s no way that Heroku is going to directly compete with AWS, or is there a GCP. That’s just not the game we play in. We’re just not that game.”

Even a company as large as Amazon can only expand so much at a time, and thus it must face tradeoffs between horizontal and vertical expansion. In fact, Layer 1 cloud providers operate at somewhat of a disadvantage in terms of offering an opinionated developer experience, because they face competitive pressure to offer a product that can fit every need a client may face. If a large client demands Kubernetes, AWS must build a Kubernetes product. Layer 2 cloud providers like Heroku can afford to be more selective because the needs of their target clients tend to be more clearly defined. From Andy Appleton:

“ I think there’s…80% of applications which have very similar requirements and kind of operate in the same way. A big goal is to serve those 80% kind of very closely so that… as close to being no operational burden on the team is possible. Then when you get customers who have much more specific requirements to try and build the Heroku platform in such a way that there’s these escape hatches or ways to dropdown a level and let them do the thing that they want to do.”

While the big tech companies may find competitive value in the adoption of the newest, shiniest cloud technologies, the operational burden of extra complexity in a tech stack may be unnecessary for organizations without the need or resources to adopt a highly configured approach. 98.2% of firms in the United States have fewer than 100 employees, and it can be safely assumed that most of those firms cannot devote all 100 employees to configuring load balancers. A comparison may be made to Spring Boot, a “convention over configuration” web framework for Java, which has become the top web framework for Java developers due, in a large part, to ease of use. As mentioned before, streamlining does not equal simplifying, and Heroku maintains a significant competitive moat due both to the technological challenge of wrangling cloud services and the business challenge of maintaining margins. Heroku charges a premium over what one would pay to use AWS alone, and maintaining the value of that margin is a primary business objective. 

Despite the business and technological challenges of operating in the gap between clients and Layer 1 cloud providers, Heroku’s products offer a sustainable value proposition to developers and tech companies who seek to reduce overhead, pass off complexity, and allocate resources most effectively towards their core business. As Jon Daniel put it:

“…I don’t have to think about building a web framework. I can just focus on building an application that provides value to my business, and using Heroku is similar….You just focus on building your apps.”

Danny Seymour

Santa Fe, New Mexico
Education: MBA, Finance and Public Policy, University of New Mexico

Danny is a Santa Fe-based developer who works as a Junior Consultant at Rural Sourcing.

Software Weekly

Software Weekly

Subscribe to Software Weekly, a curated weekly newsletter featuring the best and newest from the software engineering community.