I’m Still Iffy About “True Serverless” Infrastructure

I’m Still Iffy About “True Serverless” Infrastructure


4 min read

It feels like it was just yesterday that the world moved away from VMs to containers, and now everything from Netflix to a small student portfolio website runs on kubernetes apparently. But now, it seems like some in the tech space is ready to drop containers for the next brand new technology, serverless.

Serverless technologies come in many forms, like AWS Lambda (which is just containers in a trenchcoat) or Google Firebase (more as a BaaS or Backend-as-a-service). But what i’m more focusing on is the use of V8 Isolates as outlined by Cloudflare for use in it’s Cloudflare Workers solution.

Cloudflare says that it’s V8 isolates is the first serverless solution that doesn’t rely on containers or VMs for multitenant isolation, unlike Lambda that relies on microvm spawns by Firecracker and Firebase that runs on on Kubernetes.

Cloudflare Workers is interesting from a financial perspective as they’ve made a platform that runs faster than Lambda at 1/3rd the cost by compiling any language down to WASM and running it inside V8 isolates. In some cases like Java, they even compile the whole JVM into V8 isolates. And it also eliminates coldstarts and getting billed for I/O wait times.

But as someone who is in the security and infrastructure side of things, i remain unconvinced.

The thing with V8 isolates is that you have to reinvent alot of things to get good security and resource isolation. V8 is a runtime engine that was not build with system boundaries in mind. This doesn’t mean escaping V8 isolates is trivial, but it’s common without proper security boundaries with solutions such as gVisor. Cloudflare is promising bold things for a piece of software that wasn’t written to do what they’re using it for.

Frankly this is just a repeat of a cycle for infrastructure technologies. UNIX went through an era of thinking they can build multitenant processes that amortized resources across customers, Java went through an era of thinking they can build multitenant VMs with a bunch of solutions for protections and permissions, and Docker went through an era of thinking they can build multitenant systems that amortized the cost of resources across customers.

The V8 security team recommends running untrusted code in separate processes, but Cloudflare doesn't do that. From what Kenton Varda (Principal Engineer for the Cloudflare Workers project) said at HN, they had already committed to their architecture before the V8 recommendation was made.

It seems like they thought through a lot of possible attacks and created some clever mitigations, and ultimately decided "this is fine" by stating the V8s team recommendation are based on browser-specific environments and don’t apply to them. All of this sounds like a very regressive take on process isolation.

The issue is that a smart exploitation of a V8 bug can be super hard to detect, thus im not exactly sure how one can detect the exfiltration of customer secrets. Since 2019 there has been 10 in-the-wild zero days for V8, all of which can break the isolation measures of Cloudflare Workers.

To me the whole idea screams as “how come nobody realized you get better performance and pricing if you ignore security and compliance issues?”. Good luck ever certifiying for PCI DSS or Common Criteria. Will this be the future of computing? Probably, but definitely not on it’s current form.

A compromise i’ve seen on this technology is AWS’s CloudFront Functions, which seems specifically targeted at people who are itching to use Cloudflare Workers. Different than Cloudflare Workers, Node, or Deno, it uses an ECMAScript 5.1-compliant JavaScript engine that doesn’t make use of V8 and the system isolates every single function into its own process.

Unfortunately, it’s only a way to deterministically manipulate headers at the edge with a programming language you are used to (kinda like a way to implement glue code) and not a full on product to compete with Workers, the system doesn’t have any I/O or networking but i bet if the idea of isolates catches on AWS will find a way to copy it to hell, just not with the same methodology.

Cloudflare Workers is frankly just one V8 JIT bug away from getting all workers that share a single process compromised. Most of the CVEs affecting V8 can be turned into arbitrary read/writes so ASLR won’t help. If you host a number of workers in a single process, an attacker will never touch the OS layer and does not care about seccomp. The V8 isolation layer is as secure as an Chrome renderer process and historically that‘s not a good thing.

The argument that process isolation is a hurdle towards savings doesn’t seem to line up as many are working to make it cheaper. Case in point site isolation in Android Chrome wasn’t implemented until 2019 due to the performance penalties incurred by it, the solution wasn’t to abandon isolation alltogether but actively working to lower the performance penalties of process isolation.