About SPIFFE and SPIRE
If you're reading this, you likely already know what SPIFFE and SPIRE are. But in case you don't here is a really short summary: SPIFFE (Secure Production Identity Framework for Everyone) is a specification and SPIRE (SPIFFE Runtime Environment) an implementation of that specification for securely issuing identities to workloads running in different compute environments, and for managing these identities (such as refreshing, revoking, etc.). Why is it useful? Well, it lets your services, such as pods running in Kubernetes, to have their own certificates and signed JWTs, which are automatically refreshed, etc. and using which they can authenticate themselves to other services, and communicate securely with them. For example, the certificates could be used to create mTLS connections with other workloads, or signed tokens could be used as a proof-of-possession for authentication.
One key problem of securely issuing identities is the security of the initial handshake, for the initial request asking for identity. SPIRE solves this in a novel way using agents that are capable of querying the compute environment about the workloads requesting identities, and then issuing the identities only if these workloads satisfy certain criteria. By keeping the agents local to the node where the workloads run, concerns about the initial secrets are addressed. Of course there is a lot more to it, and the right place to head to for more details is here.
Motivation
I had trouble wrapping my head around exactly what was going on and I still have many questions about it, but I figured that the best way to learn about this was to try it out. The purpose of this post was to document the steps for doing so, focusing on deploying SPIRE for workloads on a Kubernetes cluster. It's not particularly hard to do this by following the official documentation, but this is a more linear version of it, focusing on a specific and commonly useful scenario. So I hope to make it a wee bit easier with this post.
Architecture
We will build a setup that can serve one or more Kubernetes clusters. In order to support this, we would use a relatively recent Kubernetes version (1.22+) that supports projected service account tokens (PSATs).
We will deploy a single SPIRE server running outside the k8s cluster(s). In each cluster (we will use only one), we will deploy a SPIRE agent daemonset. Each SPIRE agent instance would connect to the SPIRE server (either securely or insecurely) at bootstrap and receive the node identity. The SPIRE server and each SPIRE agent instance would securely connect to the kube API of each k8s cluster to query k8s metadata about nodes and workloads.
Deploying the SPIRE Server
Deploy the server on a Linux box which can access your k8s cluster API. For example, I am using my Ubuntu laptop which acts as the host for my k8s VMs.
Create the spire user and spire group.
$ sudo groupadd spire
$ sudo useradd -g spire -s /bin/false -M -r spire
Download the binary bundle and copy it to /opt/spire.
$ wget https://github.com/spiffe/spire/releases/download/v1.6.1/spire-1.6.1-linux-x86_64-glibc.tar.gz
$ tar xfz spire-1.6.1-linux-x86_64-glibc.tar.gz
$ sudo cp -r spire-1.6.1/ /opt/spire/
$ sudo find /opt/spire -type d -exec chmod 755 {} \;
$ sudo chmod 755 /opt/spire/bin/spire-server
$ sudo chown spire:spire /opt/spire/conf/server/server.conf
$ sudo ln -s /opt/spire/bin/spire-server /usr/bin/spire-server
Edit the server configuration present at /opt/spire/conf/server/server.conf thus.
server {
bind_address = "192.168.219.1"
bind_port = "8081"
trust_domain = "everett.host"
data_dir = "/var/opt/spire/data/server"
log_level = "DEBUG"
log_file = "/var/opt/spire/log/server.log"
ca_ttl = "168h"
default_x509_svid_ttl = "28h"
}
Set the bind_address to an address that is accessible from the k8s nodes. In my case, it is the VirtualBox host network IP of the Ubuntu host, which happens to be the gateway IP for that network. Keep the bind_port as 8081 or a different port above 1023 if you know 8081 conflicts with another application.
Set the trust_domain to a unique string - it need not be DNS resolvable. In my case, the hostname of the Ubuntu laptop where the SPIRE server is running is everett, so I set trust_domain to everett.host. Also ensure that ca_ttl is at least six times the value in default_x509_svid_ttl.
The data_dir directory specifies a directory under which most application data would be persisted. Likewise, the log_file directive specifies the log file name. I made sure I ran the following commands to make these directories accessible.
$ sudo mkdir -p /var/opt/spire/data
$ sudo mkdir -p /var/opt/spire/log
$ sudo chown -R spire:spire /var/opt/spire/
In the plugins section of the same file, update the DataStore, KeyManager, UpstreamAuthority, and NodeAttestor plugins.
For
the SQL data store, keep the database_type as the default sqlite3, and
update the connection string to point to the location under data_dir
where the data files would be stored, as shown below.
plugins {
DataStore "sql" {
plugin_data {
database_type = "sqlite3"
connection_string = "/var/opt/spire/data/server/datastore.sqlite3"
}
}
For the key manager disk plugin, specify the location where the server private keys would be kept as shown. Here we use an unencrypted directory store for our purposes, but for better security one should use more secure secret stores or key management systems.
KeyManager "disk" {
plugin_data {
keys_path = "/var/opt/spire/data/server/keys.json"
}
}
Add the UpstreamAuthority stanza if it's not present, or configure it as shown below. It configures the X509 certificate and private key used by the server to issue certificates. You need to know what you're doing, but you can read the nifty script at this site, then tweak it as needed and use that to generate a root cert and key pair, and a leaf cert and key from it. Rename the root cert and key to bootstrap.crt and bootstrap.key, copy them over to /var/opt/spire/data/server, and remember to update their permissions so that they are accessible by the spire user and spire group.
UpstreamAuthority "disk" {
plugin_data {
key_file_path = "/var/opt/spire/data/server/bootstrap.key"
cert_file_path = "/var/opt/spire/data/server/bootstrap.crt"
}
}
Finally, for each k8s cluster that this SPIRE server needs to serve, ensure a section is available as shown in bold below. The kube_config_file points to the location of the k8s config used to access the cluster's kube-api. Ensure that this file is copied from the k8s cluster to the location listed here. The cluster identifier in this case is e1, which is arbitrarily chosen - you can give it any name but make sure to use the same name to refer to it elsewhere too. The service_account_allow_list lists the service accounts from the e1 cluster that are allowed to connect to the SPIRE server. The SPIRE server would validate the service account token by using the k8s Token Review API on the cluster e1.
NodeAttestor "k8s_psat" {
plugin_data {
clusters = {
"e1" = {
service_account_allow_list = ["spire:spire-agent"]
kube_config_file = "/home/amukher1/.kube/config"
}
}
}
}
Ensure that the spire user has read access to the path listed for kube_config_file.
Next, create a systemd module for automatically starting and stopping the SPIRE server on this node. Create the file /etc/systemd/system/spire-server.service and set its content to the following.
[Unit]
Description=SPIRE Server
[Service]
User=spire
Group=spire
ExecStart=/usr/bin/spire-server run -config /opt/spire/conf/server/server.conf
[Install]
WantedBy=multi-user.target
Then enable and start the service, and check its status using:
$ sudo systemctl daemon-reload
$ sudo systemctl enable --now spire-server
$ sudo systemctl status spire-server
In case you are running this on a host that runs guest VMs in VirtualBox, and the SPIRE server listens on an IP of the host-only network that the VMs too are a part of, then that network would not be up till you bring up your first VM. Till such time, the SPIRE server could fail to start as it is unable to bind to an address off of that network. This is the reason I have not configured the service to restart automatically. Instead, you can just manually start it using the following command, once you've started the first VM:
$ systemctl start spire-server
Deploying the SPIRE Agent
The agent must be deployed as a daemonset on each k8s cluster that you want to manage via this server. We would need certain manifest yamls for deploying the agent. We could get this from the source bundle, downloaded from the page here.
$ wget https://github.com/spiffe/spire/archive/v1.6.1.tar.gz
$ tar -xf v1.6.1.tar.gz spire-1.6.1/test/integration/suites/k8s/conf/agent/spire-agent.yaml --strip-components=7
Then edit the file spire-agent.yaml as below.
In the spire-agent ConfigMap manifest inside this file, edit the contents of agent.conf, setting the following keys:
server_address = "<SPIRE_server_addr>"
This should be the IP or the FQDN where the SPIRE server is accessible - typically the same as the bind_address in the server config, or an FQDN resolving to it.
Also set the trust_domain to the same value as set for the server:
trust_domain = "everett.host"
Make a note of the trust_bundle_path attribute. This is the location within the SPIRE agent pod where the CA cert bundle of the SPIRE server should be mounted. It is okay to keep it set to its default value of /run/spire/bundle/bundle.crt. On first run you may want to comment out this attribute and instead add the following.
insecure_bootstrap = true
Within the NodeAttestor stanza, set the cluster attribute to e1.
NodeAttestor "k8s_psat" {
plugin_data {
cluster = "e1"
}
}
In the WorkloadAttestor stanza, set the directive skip_kubelet_verification to false, and set the kubelet_ca_path attribute to the location CA cert for this k8s cluster as shown below.
WorkloadAttestor "k8s" {
plugin_data {
...
# skip_kubelet_verification = false
kubelet_ca_path = "/run/spire/bundle/kubelet-ca.crt"
}
We will ensure that the k8s cluster CA cert is mounted at the location pointed at by kubelet_ca_path, via a ConfigMap.
Further down in the manifest, edit the containers section of the DaemonSet spec, updating the SPIRE agent image location, and image pull policy.
containers:
- name: spire-agent
image: ghcr.io/spiffe/spire-agent:1.6.1
imagePullPolicy: IfNotPresent
Copy the CA cert of your k8s cluster from /etc/kubernetes/pki/ca.crt to the local directory, naming the target file kubelet-ca.crt.
If you did not set insecure_bootstrap to true earlier, then retrieve the CA cert bundle for the SPIRE server that was set aside, and copy it to some location from where you can create ConfigMaps on this cluster. Rename the file to bundle.crt.
Then run the following command, include the bundle.crt only if you copied it.
$ kubectl create configmap spire-bundle -n spire --from-file=kubelet-ca.crt --from-file=bundle.crt
Finally, apply this edited manifest on your k8s cluster.
$ kubectl apply -f spire-agent.yaml
If all goes well, you should have a working SPIRE installation on your k8s cluster. You can verify that the SPIRE agent daemonset has pods running on each worker node of your cluster by running the following command.
$ kubectl get pods -n spire
Make sure that the pods are running and ready. Also run the following commands on the SPIRE server to verify that the SPIRE agents on your nodes have been attested and received SPIFFE ids. You can get agent SPIFFE ids from the output of the first command.
$ spire-server agent list
$ spire-server agent show -spiffeID <spiffe_ID>
Cleaning up
While experimenting with the setup, one would likely need to clean up and recreate the setup several times. When doing that, it's good to follow a certain discipline. With all nodes of the cluster up, run the following commands.
$ kubectl delete -f spire-agent.yaml -n spire
$ kubectl delete configmap spire-bundle -n spire
$ kubectl delete ns spire
$ kubectl get all,configmap,sa -n spire
Other than the default service account and a ConfigMap called kube-root-ca.crt, all other resources should be deleted. Still check for any stray pods:
$ kubectl get pods -n spire
Using SPIRE
In this section, we shall see how to use SPIRE to have identities issued to workloads running within your Kubernetes cluster.
Architecture
We would deploy a pod on Kubernetes to fetch its identity bundle from the SPIRE agent running locally on that node. It would connect to the agent over a Unix domain socket mounted from a host path. We must also create registration entries for the workloads. Typically we would want to do this by defining criteria for choosing a workload, using selectors. We want to create the rules such that given a pod running a particular image X and with a particular label app=Y, we would always issue it the same identity no matter which worker node it runs on. To do this, the parent spiffe ID of the registration entry cannot be that of a single worker node, but of an alias to all the worker nodes. The following section describes these in detail.
Process
Run the command below on the server to create a node alias SPIFFE id that applies to all worker nodes of the cluster. This will allow us to create registration entries for workloads that would give them the same identity based on their image and a label, irrespective of which cluster node they run on.
/opt/spire/bin/spire-server entry create -node -spiffeID spiffe://everett.host/ns/spire/sa/spire-agent/cluster/e1 -selector k8s_psat:cluster:e1 -selector k8s_psat:agent_ns:spire -selector k8s_psat:agent_sa:spire-agent
Next, create a binary using the following Go code. This binary fetches its identity and bundles from the SPIRE agent running locally on a worker.
package main
import (
"context"
"fmt"
"log"
"net/http"
"os"
"strings"
"time"
"github.com/spiffe/go-spiffe/v2/spiffeid"
"github.com/spiffe/go-spiffe/v2/svid/jwtsvid"
"github.com/spiffe/go-spiffe/v2/workloadapi"
)
const (
socketPath = "unix:///tmp/spire-agent/api.sock"
)
func main() {
ctx := context.Background()
for err := run(ctx); ; {
if err != nil {
log.Fatal(err)
}
time.Sleep(60 * time.Second)
}
}
func run(ctx context.Context) error {
// Set a timeout to prevent the request from hanging if this workload is not properly registered in SPIRE.
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
client := workloadapi.WithClientOptions(workloadapi.WithAddr(socketPath))
// Create an X509Source struct to fetch the trust bundle as needed to verify the X509-SVID presented by the server.
x509Source, err := workloadapi.NewX509Source(ctx, client)
if err != nil {
fmt.Printf("unable to create X509Source: %v", err)
return fmt.Errorf("unable to create X509Source: %w", err)
}
defer x509Source.Close()
fmt.Printf("Received trust budle: %v", x509Source)
serverID := spiffeid.RequireFromString("spiffe://example.org/server")
// By default, this example uses the server's SPIFFE ID as the audience.
// It doesn't have to be a SPIFFE ID as long as it follows the JWT-SVID guidelines (https://github.com/spiffe/spiffe/blob/main/standards/JWT-SVID.md#32-audience)
audience := serverID.String()
args := os.Args
if len(args) >= 2 {
audience = args[1]
}
// Create a JWTSource to fetch JWT-SVIDs
jwtSource, err := workloadapi.NewJWTSource(ctx, client)
if err != nil {
fmt.Printf("unable to create JWTSource: %v\n", err)
return fmt.Errorf("unable to create JWTSource: %w", err)
}
defer jwtSource.Close()
// Fetch a JWT-SVID and set the `Authorization` header.
// Alternatively, it is possible to fetch the JWT-SVID using `workloadapi.FetchJWTSVID`.
svid, err := jwtSource.FetchJWTSVID(ctx, jwtsvid.Params{
Audience: audience,
})
if err != nil {
fmt.Printf("unable to fetch SVID: %v\n", err)
return fmt.Errorf("unable to fetch SVID: %w", err)
}
fmt.Printf("Received JWT svid: %v", svid)
return nil
}
I named this binary id-client, and created a docker image tagged amukher1/id-client and pushed it to DockerHub. You can use a name of your choice. Deploy this binary with the following manifest:
apiVersion: v1
kind: Pod
metadata:
name: id-client
labels:
app: id-client
spec:
containers:
- name: id-client
image: amukher1/id-client
volumeMounts:
- name: spire-agent-socket
mountPath: /tmp/spire-agent
readOnly: false
volumes:
- name: spire-agent-socket
hostPath:
path: /run/spire/agent-sockets
type: DirectoryOrCreate
Make a note of the image tag of the container image. I used the following command and copied the first value inside the RepoDigests array:
docker inspect amukher1/id-client
Now create a registration entry for a workload matching this binary, using the node alias as the parent SPIFFE id.
/opt/spire/bin/spire-server entry create -spiffeID spiffe://everett.host/image/id-client -parentID spiffe://everett.host/ns/spire/sa/spire-agent/cluster/e1 -selector k8s:pod-image:docker.io/<image tag> -selector k8s:pod-label:app:id-client
With the above steps, you should be able to fetch x509 certificates as well JWT tokens.
Read more!
