This article, like its predecessor Quickly Building Envoy Proxy, is an attempt to document what should have been widely documented but isn't. Serious open source communities sometimes function in an elitist way to perhaps keep the entry bar high. Maybe that's why they consciously avoid being too focused on documenting the basic mechanisms that people need in order to work with the code base. But for commoners like me this lack of documentation becomes motivation to figure things out and write about them with the hope that someone else finds it easier.
If you're building Envoy from source code, then maybe you've got reason to modify Envoy source code, and one of the most common things that people need to do with the Envoy code base is to write new filters.
What are filters?
You possibly already know this, but filters are pluggable code that you can run within Envoy to customize how you process incoming requests, what responses you send, etc. When a request enters Envoy, it enters through a listener (a listener port). This request then goes through a filter chain - which might be chosen from among multiple filter chains based on some criteria - like the source or destination address or port. Each filter chain is a sequence of filters - logic that runs on the incoming request and perhaps modifies it or determines whether it would be processed further and how.
Filters can be written in C++, Lua, Wasm (and therefore any language that compiles to Wasm), and apparently in Go and Rust too. I know precious little about the last ones but they sound interesting. Lua filters are quite limited in many ways and hence I decided not to focus on them. Native filters in C++ seem to be adequate for most purposes so this post is about them.
What are the different types of filters?
There are different kinds of filters. Listener filters are useful if some actions need to be performed while accepting connections. Then there are network filters which operate at the L4 layer on requests and responses. Two such filters are HTTP Connection Manager (HCM) which is used to process http traffic, and TCP Proxy, which is used to route generic TCP traffic. Within HCM, it is possible to load a further class of filters which are aware of the http protocol - these are called HTTP filters and operate at the L7 layer. In this article, we will focus on L4 layer filters or network filters.
Where does one write filters?
Filter writing seems to have gotten a wee bit easier over the successive Envoy versions and seemed somewhat agreeable when I tried it on version 1.27.x. One can write a filter in a separate repo and include all of the Envoy code as a sub-module. However, I wanted to write an in-tree filter - just like the several filters already part of the code base.
How to write your first filter?
We will write a network filter which will log the destination IP of a request as it was received, and then forward the request to some destination based on a mapping of destination IPs to actual target addresses. This is actually useful functionality which isn't available out of the box from Envoy, but requires just a small amount of filter code to get going. We will call this filter the Address Mapper filter. So what do we need?
The config proto
Most filters need some configuration. In our case, our filter would take a map of IP addresses to cluster names - Envoy clusters representing one or more endpoints where the traffic could be sent. So essentially, we are looking for a map<string, string> as input to the filter. However, to make things a little bit more type-safe, Envoy needs to know exactly what is the type of the input config. So we must define a protobuf message describing this input config. We create this under api/envoy/extensions/filters/network/address_mapper/v3. The filter would be a network filter, and it would be called address_mapper. So we created a directory under api/envoy/extensions/filters/network, by the name address_mapper. One further sub-directory under it, v3, holds the actual protos. v3 represents the current generation of Envoy's config API - v1 and v2 are obsolete versions. The proto file, address_mapper.proto, is placed under v3 and has the following content.
syntax = "proto3";
package envoy.extensions.filters.network.address_mapper.v3;
import "udpa/annotations/status.proto";
option java_package = "io.envoyproxy.envoy.extensions.filters.network.address_mapper.v3";
option java_outer_classname = "AddressMapper";
option java_multiple_files = true;
option go_package = "github.com/envoyproxy/go-control-plane/envoy/extensions/filters/network/address_mapper/v3;address_mapperv3";
option (udpa.annotations.file_status).package_version_status = ACTIVE;
// [#protodoc-title: Address mapper]
// Connection limit :ref:`configuration overview <config_network_filters_connection_limit>
// [#extension: envoy.filters.network.address_mapper]
message AddressMapper {
// address_map is expected to contain a 1:1 mapping of
// IP addresses to other IP addresses or FQDNs.
map<string, string> address_map = 1;
}
We must also create a Bazel BUILD file in the same directory, and that's the limit of what I am qualified to say about these abominations used to build the whole Envoy binary and its various parts. So
# DO NOT EDIT. This file is generated by tools/proto_format/proto_sync.py.
load("@envoy_api//bazel:api_build_system.bzl", "api_proto_package")
licenses(["notice"]) # Apache 2
api_proto_package(
deps = ["@com_github_cncf_udpa//udpa/annotations:pkg"],
)
If at this time you want to interject profanities about Bazel (or at any other time), you know it is wrong.
Anyway, so you need to link your proto up to the build chain. So you have to make entries inside api/BUILD and api/versioning/BUILD. Make the following entry under the v3_protos library in api/BUILD, and under active_protos in api/versioning/BUILD.
"//envoy/extensions/filters/network/address_mapper/v3:pkg",
We must also create a type URL that Envoy would recognize and instantiate the config message of the correct type. To do this we create an entry for the AddressMapper message inside source/extensions/extensions_metadata.yaml.
envoy.filters.network.address_mapper:
categories:
- envoy.filters.network
security_posture: robust_to_untrusted_downstream_and_upstream
status: stable
type_urls:
- envoy.extensions.filters.network.address_mapper.v3.AddressMapper
This introduces the new filter, and a type URL for the config proto on the last line. We must also tell Bazel where the source code for the new filter is present. To do this we edit source/extensions/extensions_build_config.bzl creating the following entry in the network filters section:
"envoy.filters.network.address_mapper": "//source/extensions/filters/network/address_mapper:config",
Envoy must also recognize the fully-qualified string representing the new network filter we are going to create. Because it is a network filter, we add it in source/extensions/filters/network/well_known_names.h. Inside the class NetworkFilterNameValues, we add the following const member.
// Address mapper filter
const std::string AddressMapper = "envoy.filters.network.address_mapper";
The filter logic
We must add the filter logic somewhere. To do this, we create a new directory called address_mapper under source/extensions/filters/network/address_mapper/. We first add the AddressMapperFilter filter class definition in address_mapper.h and also an AddressMapperConfig class which wraps the config message passed via the Envoy config. These are all inside the Envoy::Extensions::NetworkFilters::AddressMapper namespace.
class AddressMapperConfig {
public:
AddressMapperConfig(const FilterConfig& proto_config);
absl::string_view getMappedAddress(const absl::string_view& addr) const;
private:
absl::flat_hash_map<std::string, std::string> addr_map_;
};
The filter takes a shared_ptr to the above config class.
using AddressMapperConfigPtr = std::shared_ptr<AddressMapperConfig>;
class AddressMapperFilter : public Network::ReadFilter, Logger::Loggable<Logger::Id::filter> {
public:
AddressMapperFilter(AddressMapperConfigPtr config);
// Network::ReadFilter
Network::FilterStatus onData(Buffer::Instance&, bool) override {
return Network::FilterStatus::Continue;
}
Network::FilterStatus onNewConnection() override;
void initializeReadFilterCallbacks(
Network::ReadFilterCallbacks& callbacks) override {
read_callbacks_ = &callbacks;
}
private:
Network::ReadFilterCallbacks* read_callbacks_{};
AddressMapperConfigPtr config_;
};
The implementation of the onNewConnection method is in the address_mapper.cc file. For example, we can get the original destination address like this.
Network::Address::InstanceConstSharedPtr dest_addr =
Network::Utility::getOriginalDst(const_cast<Network::Socket&>(read_callbacks_->socket()));
We can then map this address to the target cluster, etc.
Someone has to instantiate this filter and pass it the correct type of argument (AddressMapperConfigPtr). That responsibility falls with the glue code or filter factory, which we look at next.
The glue code
We define the config factory (AddressMapperConfigFactory) class inside the config.h header in the filter directory. These are all inside the Envoy::Extensions::NetworkFilters::AddressMapper namespace.
class AddressMapperConfigFactory
: public Common::FactoryBase<
envoy::extensions::filters::network::address_mapper::v3::AddressMapper> {
public:
AddressMapperConfigFactory() : FactoryBase(NetworkFilterNames::get().AddressMapper) {}
/* ProtobufTypes::MessagePtr createEmptyConfigProto() override; */
std::string name() const override { return NetworkFilterNames::get().AddressMapper; }
private:
Network::FilterFactoryCb createFilterFactoryFromProtoTyped(
const envoy::extensions::filters::network::address_mapper::v3::AddressMapper& proto_config,
Server::Configuration::FactoryContext&) override;
};
We now add the implementation for createFilterFactoryFromProtoTyped, which is the entry point for filter instantiation.
Network::FilterFactoryCb AddressMapperConfigFactory::createFilterFactoryFromProtoTyped(
const envoy::extensions::filters::network::address_mapper::v3::AddressMapper& proto_config,
Server::Configuration::FactoryContext&) {
AddressMapperConfigPtr filter_config = std::make_shared<AddressMapperConfig>(proto_config);
return [filter_config](Network::FilterManager& filter_manager) -> void {
filter_manager.addReadFilter(std::make_shared<AddressMapperFilter>(filter_config));
};
}
Given the protobuf input from the configuration, this code gives back an instance of the actual filter initialized with this config.
How to compile your first filter?
You need to ensure that your filter code is included in the BUILD. Create the BUILD file in your filter directory with the following content.
load(
"//bazel:envoy_build_system.bzl",
"envoy_cc_extension",
"envoy_cc_library",
"envoy_extension_package",
)
licenses(["notice"]) # Apache 2
envoy_extension_package()
envoy_cc_library(
name = "address_mapper",
srcs = ["address_mapper.cc"],
hdrs = ["address_mapper.h"],
deps = [
"//envoy/network:connection_interface",
"//envoy/network:filter_interface",
"//source/common/common:assert_lib",
"//source/common/common:minimal_logger_lib",
"//source/common/tcp_proxy",
"//source/common/protobuf:utility_lib",
"//source/common/network:utility_lib",
"@envoy_api//envoy/extensions/filters/network/address_mapper/v3:pkg_cc_proto",
],
alwayslink = 1,
)
envoy_cc_extension(
name = "config",
srcs = ["config.cc"],
hdrs = ["config.h"],
deps = [
":address_mapper",
"//envoy/registry",
"//envoy/server:filter_config_interface",
"//source/extensions/filters/network/common:factory_base_lib",
"//source/extensions/filters/network:well_known_names",
"@envoy_api//envoy/extensions/filters/network/address_mapper/v3:pkg_cc_proto",
],
)
The exact dependencies listed depend on what you need to call from within your filter code (something we haven't yet shown). For example, the protobuf utility_lib or network utility_lib are listed, as is the network connection_interface.
The previous article in this series already shows how to build Envoy. That is all you need to do to build Envoy with this filter enabled. One handy ability is to build Envoy with debug symbols. This is quite easy:
bazel build envoy -c debug
The binary is created under ./bazel-bin/source/exe/envoy-static.
Configuring Envoy to run your filter
In our case, we want the filter to be put before a TCP Proxy filter. So the config should look like this:
- name: envoy.filters.network.address_mapper
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.address_mapper.v3.AddressMapper
address_map:
"169.254.1.2": "cluster_1"
"169.254.1.3": "cluster_2"
- name: envoy.filters.network.tcp_proxy
typed_config:
...
The assumption is that the clusters cluster_1 and cluster_2 are separately defined elsewhere in the config. Our filter checks if the original destination IP of the incoming request matches the IPs listed in the address map and if it does, then it sets a connection streamInfo filter-state metadata (TcpProxy::PerConnectionCluster) that tells the ensuing TCP proxy filter to forward the request to the mapped cluster.
Conclusion
There are lots of gaps in this article (because it was hurriedly written), but refer to existing filter code to fill those gaps in. It should be fairly straightforward.
Read more!
