Envoy, Istio & Wasm

Recently I took the dive into the world of Wasm (WebAssembly) plugins, particularly in the context of Envoy and Istio.

Let’s start off by talking about Wasm, or WebAssembly. Wasm is a compact toolkit for stack-based virtual machines. It’s fast and it can integrate seamlessly with both client and server applications.

Where does Envoy and Istio fit into this? Envoy is an open-source edge and service proxy designed for cloud-native applications. It can handle directing traffic between your services. On the other hand, Istio is a service mesh, utilizing Envoy as its data plane to control your mesh. It’s Istio that orchestrates the microservices network of your applications, giving you plenty of power in your Kubernetes cluster.

Wasm’s Power

Wasm is an assembly language for a conceptual machine, not an actual physical machine. This means it’s designed to be efficient for any machine to execute, making it great to run just about anywhere.

Wasm is incredibly powerful as an extension tool, particularly for systems like Envoy. With Wasm, you can add functionality to Envoy without needing to understand the intricate details of its C++ codebase. This could range from manipulating HTTP headers to more complex traffic management and telemetry gathering operations.

Wasm modules run in a sandboxed environment, making them secure and isolated from the main Envoy process. They can be written in multiple languages, but we’re particularly interested in Golang as it’s what I use in my day-to-day. We can leverage the proxy-wasm-go-sdk library to help with this.

It’s worth noting that TinyGo, rather than standard Golang, must be used. This is primarily because standard Golang cannot build Wasm binaries that run outside of a web browser at this time.

Build It

Let’s build a simple Wasm plugin that adds a specific HTTP header to all incoming requests to enable tracking. We’ll use the proxy-wasm-go-sdk library and TinyGo. Remember this is bare bones!

In this code, we’ve set our headerName and headerValue in the OnPluginStart method to “X-Request-Id” and “request-tracking” respectively. These are the values you’ll specify in the pluginConfig of the WasmPlugin resource later.

package main

import (
	"github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm"
	"github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm/types"
)

func main() {
	proxywasm.SetVMContext(&vmContext{})
}

type vmContext struct {
	types.DefaultVMContext
}

func (*vmContext) NewPluginContext(contextID uint32) types.PluginContext {
	return &pluginContext{}
}

type pluginContext struct {
	types.DefaultPluginContext
	headerName string
}

func (p *pluginContext) NewHttpContext(contextID uint32) types.HttpContext {
	return &httpHeaders{
		contextID:  contextID,
		headerName: p.headerName,
	}
}

func (p *pluginContext) OnPluginStart(pluginConfigurationSize int) types.OnPluginStartStatus {
	data, err := proxywasm.GetPluginConfiguration()
	if data == nil {
		proxywasm.LogInfof("no configuration provided for the plugin")
		return types.OnPluginStartStatusOK
	}

	if err != nil {
		proxywasm.LogCriticalf("error reading plugin configuration: %v", err)
		return types.OnPluginStartStatusFailed
	}

	p.headerName = string(data)
	proxywasm.LogInfof("header from config: %s", p.headerName)

	return types.OnPluginStartStatusOK
}

type httpHeaders struct {
	types.DefaultHttpContext
	contextID  uint32
	headerName string
}

func (ctx *httpHeaders) OnHttpRequestHeaders(numHeaders int, endOfStream bool) types.Action {
	err := proxywasm.AddHttpRequestHeader(ctx.headerName, "request-tracking")
	if err != nil {
		proxywasm.LogCriticalf("failed to add request header: %s", ctx.headerName)
	}

	return types.ActionContinue
}

func (ctx *httpHeaders) OnHttpStreamDone() {
	proxywasm.LogInfof("%d finished", ctx.contextID)
}

The above Go code is designed to implement a Wasm module that uses the proxy-wasm-go-sdk library.

Here’s the flow

Initialization: The code starts by defining the main function and initializing the Wasm virtual machine with a vmContext that is a struct embedding the DefaultVMContext. It is set by calling proxywasm.SetVMContext.

NewPluginContext: The NewPluginContext method in vmContext is overridden to return a new instance of the pluginContext struct. This struct also embeds the DefaultPluginContext and adds a new field headerName, which is used to store the name of the HTTP header to be added.

OnPluginStart: When the plugin starts, the OnPluginStart method of the pluginContext is called. It tries to read the plugin configuration using proxywasm.GetPluginConfiguration(). If the configuration is successfully read, it is stored in the headerName field of the pluginContext.

NewHttpContext: The NewHttpContext method of the pluginContext struct is overridden to return a new instance of the httpHeaders struct. This struct holds the contextID and headerName for each HTTP request.

OnHttpRequestHeaders: This method is called for each HTTP request. It adds an HTTP header with the name stored in headerName and the value “request-tracking” to the request. If there is an error adding the header, it logs a critical error message.

OnHttpStreamDone: This method is called when the HTTP stream is completed. It simply logs a message indicating that the HTTP stream with a specific contextID has finished.

This Wasm module is designed to be plugged into an Istio service mesh to manipulate HTTP requests passing through the mesh’s proxies, adding a “request-tracking” header to them.

The Image

After writing your Wasm module with Go and the Proxy-Wasm SDK, it needs to be compiled into a Wasm binary and then packaged into an OCI (Open Container Initiative) image.

First, compile your Go code into a Wasm binary with TinyGo — tinygo build -o main.wasm -target wasi main.go

Next, you’ll need to build the OCI image. This can be done with a Dockerfile. Here’s an example of a minimal Dockerfile using scratch, and adds the Wasm binary into the image

FROM scratch
COPY main.wasm /plugin.wasm

Push to your repo

Deploy With Istio

So how does it work? Wasm modules get loaded into these Envoy proxies, becoming part of the network traffic flowing through them. This gives you a total control over your mesh’s behavior. Istio makes loading these modules simple.

Let’s take a look at an example

apiVersion: extensions.istio.io/v1alpha1
kind: WasmPlugin
metadata:
  name: request-tracking
  namespace: istio-ingress
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  url: oci://my-registry:5000/request-tracking/request-tracker:latest
  imagePullPolicy: IfNotPresent
  imagePullSecret: my-registry-pull-secret
  phase: HTTP_FILTER
  pluginConfig:
    tracking_header: X-Request-Id

In this configuration, we have a Wasm plugin called “request-tracking”. This plugin is loaded into the istio-ingressgateway, which is selected by the matchLabels under selector.

The Wasm module for this plugin is hosted at the URL oci://my-registry:5000/request-tracking/request-tracker:latest. The imagePullPolicy and imagePullSecret settings are used to pull this module from the registry.

The phase is set to HTTP_FILTER, meaning this plugin operates at the HTTP filter phase, which is appropriate for a plugin that manipulates HTTP headers.

Finally, in pluginConfig, we specify a tracking_header of X-Request-Id. This will be the header added by this Wasm plugin to track requests.

What’s Happening

Under the hood, when you create or update a WasmPlugin resource, Istio gets your plugin setup in the mesh. Istio’s control plane, which includes things like Istiod, pushes the new Wasm configuration to all proxies that match the selector specified in the plugin.

In our example, when you apply the request-tracking plugin, Istio retrieves the Wasm module from the specified OCI registry and dynamically injects it into the appropriate Envoy proxies. In this case, those that are labelled with istio: ingressgateway.

Once the Wasm module is loaded into Envoy, it starts functioning at the specified phase - in our case, as an HTTP_FILTER. As the traffic flows through the proxy, the plugin intercepts the requests and, in this instance, adds the X-Request-Id tracking header.

But, what if the plugin blows up and breaks? Typically, if a Wasm plugin fails during execution, it shouldn’t bring down the entire proxy or halt traffic flow. Envoy is designed to handle filter failures gracefully, isolating the effect of a misbehaving filter.

However, if a plugin is critical to request processing - such as performing an essential transformation or security check - a failure could potentially affect the requests it’s meant to process. Be careful while implementing your plugin.

Conclusion

In conclusion, Wasm integration with Envoy and Istio empowers you to extend and customize your service mesh with ease. This opens up possibilities for manipulating HTTP requests, managing traffic, and gathering telemetry data to meet your specific requirements. Remember to properly test and monitor your Wasm modules to maintain stability and performance of your mesh.

Check it out and have some fun!