AWS re:Invent 2016: Primitives, Not Frameworks

“Transformation” was the theme of Werner Vogels’ keynote this year. To match that theme, he even took the stage wearing a T-shirt printed with the Autobot insignia. This keynote is well worth watching again and again — on the surface, its core message is how we help customers become transformers of the internet-scale era through innovations in development, data, and compute. But look a little deeper, and it is really a retrospective and summary of the AWS architecture blueprint and product roadmap.

Werner split this summary into three parts: development, data, and compute.

Development

Software development in the internet era poses many challenges, especially on the operations (ops) side. Software needs to be built, deployed, monitored, dynamically scaled, kept secure, and so on.

Previously, AWS’s support for operations was mainly reflected in the scaling of its various products — from compute to storage to databases to message queues. Nearly every tool was built with internet-scale capability. However, because other operational tools were missing, users had to turn to third-party options: New Relic / Datadog / … for monitoring, Travis / Codeship / … for builds, Loggly / Splunk / … for log processing. All these assorted third-party tools added extra R&D overhead.

So naturally, Amazon extended its reach into this space. CodeCommit, CodeDeploy, CodePipeline, OpsWorks and the like that came before were merely a prelude; the services released this year fill in the operational jigsaw puzzle more completely:

• AWS OpsWorks for Chef: Chef support is now built into OpsWorks.
• Amazon EC2 Systems Manager: manages a whole range of EC2 operational activities, from software installation to system updates to resource configuration.
• AWS CodeBuild: the missing piece between CodeCommit and CodeDeploy is finally in place. This poses a direct threat to existing build tools like Travis in the ecosystem.
• Amazon X-Ray: long awaited, and at last AWS has its own APM tool. The New Relics of the world had better watch out.
• Personal Health Dashboard: this appears to be a monitoring service for your own service health, letting you understand whether your software is at a healthy level.
• AWS Shield: provides automatic DDoS protection at different layers. DDoS has become a public menace of the internet-scale era; the recent DDoS attacks on DNS services left many companies, GitHub included, badly stung.

Among these, the most exciting is X-Ray. Debugging and performance monitoring of distributed systems is a complex problem. Traditional APM can only collect data about how code is running by installing an agent on the target machine. If the target system can’t have an agent installed, there’s nothing to be done. Under AWS’s serverless architecture, Lambda, DynamoDB, and the like are all black holes for performance monitoring. But because X-Ray is Amazon’s own monitoring product, it naturally gets first-hand, trustworthy data.

Data

In this segment, Werner played a jigsaw puzzle game, splitting data analytics work into ten pieces:

• automated and reliable data ingestion: for data collection we now have S3, Kinesis, Snowball, DynamoDB Streams
• preservation of original source data: source data can be persisted through S3, EFS, EBS, DynamoDB, RDS.
• lifecycle management and cold storage: S3 / Glacier provide a complete data lifecycle — a full solution from hot data to cold data.
• metadata capture
• managing governance, security, privacy: this is an AWS strong suit — AWS Config and CloudTrail for configuration and monitoring; KMS for securing data (confidentiality, accessibility, consistency).
• self-service discovery, search, access
• managing data quality: Elastic MapReduce can process the data.
• preparing for analytics
• orchestration and job scheduling
• capturing data change

But as shown above, the puzzle was only half finished:

So the newly released AWS Glue, with its Data Catalog and ETL support, covers the other half:

And with that, we can have a complete data-processing architecture:

Compute

In 2014, AWS released Lambda as a complement to its compute architecture. Since then, the database side got the serverless DynamoDB, and the compute side got the serverless, pay-as-you-go Lambda. An application can be built with S3 web hosting (SPA) + API Gateway + Lambda + DynamoDB, without a single server, and still handle massive traffic, essentially free of any scaling problems. In a sense, Lambda is strong evidence of AWS’s courage to reinvent itself. With Lambda in use, many scenarios that previously required EC2 / ECS can be replaced, and users can accomplish more with lower cost. Especially once API Gateway, DynamoDB Streams, and S3 events could trigger Lambda, its use cases expanded enormously. The following chart compares EC2 / ECS / Lambda:

In AWS’s compute landscape, Lambda is becoming ever more important. The AWS Step Functions and Lambda@Edge released this time all but foretell the future direction of internet-scale applications.

Let’s start with Lambda@Edge. Werner barely gave this enormously important feature more than a few passing words.

AWS put Lambda inside the CDN!

AWS put Lambda inside the CDN!

AWS put Lambda inside the CDN!

The picture is just too beautiful for words! Previously, a CDN could only distribute static content, or do some preset processing on the static content to be distributed — resizing images, for example. Lambda@Edge hands the choice of further processing content to the user — the application developer. With CloudFront + Lambda@Edge, all of a sudden we have processing power deployed around the globe (excluding our great Song dynasty), right to the place closest to the user.

Let me give you an example. A client accesses a video on the CDN. Based on the user-agent, Lambda can rewrite the access to that video into another URL (say, Apple TV gets the 1080p video, iPhone gets the 480p video). For an application like TubiTV that has a dozen or so client platforms, this lets it expose just a single video URL to the outside world.

Let me give you another one. A/B testing of the client UI can be done at the CDN. Lambda can parse the cookies in the HTTP request (or the device ID in the query string, or token information in the authorization header) to determine the visitor’s identity and thereby serve the appropriate A/B testing assets.

We could even do on-the-fly transcoding of images / short videos / audio assets. For a video advertising platform, for instance, ads are fetched dynamically, and to suit different platforms, they need to be transcoded. If this operation happens at the CDN edge, rather than going the long way around CDN -> origin -> transcoding -> origin -> CDN, then the user experience on the first visit (when the ad hasn’t yet been transcoded) is greatly enhanced.

Now let’s talk about Step Functions. It takes Lambda’s power to a whole new level. Even Werner himself said this was the most important product release of this keynote!

We know that although Lambda has near-infinite scaling and charges by number of invocations, one very important reason limiting its wide adoption is that it’s hard to manage and coordinate. Each Lambda function is a loose, event-triggered microservice, and if you want multiple Lambda functions to coordinate to complete a complex state machine, development and management become extremely painful. Take the most important synchronization concerns: our current approaches are function chaining, databases, and queues. Each has its own strengths and weaknesses.

We want a better control flow to manage and synchronize Lambdas, and to visualize that flow so the processing of a state machine is clear at a glance. Step Functions offers this possibility:

Combine Step Functions with X-Ray, and the difficulties of Lambda coordination and debugging will be greatly alleviated. Once I’ve experimented with Step Functions, I’ll write in detail about its capabilities and shortcomings.

The above are my interpretations of some of the features I consider important in Werner’s keynote. The full keynote is now available on YouTube, so do give it a look.

The careful reader may be wondering by now: the title of this article is “primitives not frameworks,” and yet the article is nearly finished — how come that phrase hasn’t even come up?

That phrase comes from a blog post Werner wrote this year (allthingsdistributed.com): “10 Lessons from 10 Years of Amazon Web Services.” I even wrote an interpretation of it at the time, but felt it wasn’t high quality enough — a superfluous addition to an already fine thing — so I was too embarrassed to publish it.

Looking back at what I wrote then, there was one passage I put like this:

Primitives not frameworks — a revelation. When Google realized this, killed GAE, and raised up GCE for a counterattack, it was already too late. Strategically, IaaS offers LEGO bricks in a way that traditional IT enterprises and IT talent understand (computing, storage, database, IPC), letting these people — who still lack trust in new things — flexibly build or reshape their systems in a way that is familiar to them. This process takes longer, but it’s steady. PaaS, by contrast, tries to get there in one step, offering an excellent but rigid architecture that pulls everything together. On the surface, PaaS looks more advanced than IaaS in every way, but because it took too big a step, the learning and migration costs were too high — traditional enterprises couldn’t use it (the gap was too wide), startups couldn’t use it well, and it actually lost to the seemingly more backward IaaS. Only once Amazon had built out the primitives — storage, computing, database — and customers were handling them with ease, and then naturally shifted over to PaaS offering all sorts of high-end features, was that the right approach.

Since 2006, AWS has been steadily providing the primitives of the cloud era:

• Compute: EC2, ECS, Lambda
• Storage: S3, S3 Infrequent Access, Glacier, Snowball, Snowmobile
• Database: RDS, Aurora, DynamoDB
• Message queue: SQS
• Big data: Kinesis, Redshift, EMR, Elasticsearch Service
• Security and identity: IAM, SG, KMS, Directory Service
• Networking: Route 53, VPC, CloudFront
• Operations: CloudWatch, CodeCommit, CodeBuild, CodeDeploy, OpsWorks, X-Ray

This family has exploded from a handful of core services at the start to dozens of services today, covering every aspect of building internet-scale applications. It offers plenty of flexibility: you can build a machine learning system yourself on EC2, you can use EMR and focus on crafting your models, or even — if you want to handle image recognition or speech recognition — you can just use tools like Rekognition / Lex directly. Together they form an unmatched herd, a carrier strike group that supports one another:

And its old rivals have been left further and further behind.

This is both a good thing and a bad thing. I sincerely hope Azure and GCE can catch up with great strides, and keep this field sufficiently competitive — that is what would truly be fortunate for users, and even for the entire industry.

software is a mindset, not a skillset.