EKS vs. GKE — Security

This is another post in my series about the differences between AWS’ Elastic Kubernetes Service (EKS) and GCP’s Google Kubernetes Engine (GKE). The previous posts covered cross-cloud identity/authx, networking, Ingress and observability.

This post is focused on the differences when it comes to security between the two — both their default settings and what is possible on/across them both as part of the managed service. I also am aware that I didn’t cover network security/firewalling in the network post — and so will be covering that here as well (it felt more appropriate to cover as one of the key security topics).

When it comes to security of Kubernetes I believe the key areas are:

• Identity and access management — controlling who/what can interact with the cluster(s) and in what ways
• Controlling network traffic / firewalling—this is how traffic is controlled including both east/west (between Pods in the cluster) and north/south (in and out of the cluster)
• Configuration/Posture management — this is understanding whether the cluster (Control Plane, Nodes and Workloads) is configured securely and the extent to which that secure configuration is enforced with tools such as admission controllers
• Vulnerability/Supply-chain management — this is recognizing when any of your cluster components, or the workloads that are running on top of them, have known critical vulnerabilities that need to be patched
• Runtime threat detection — this is being alerted for suspicious behaviors that might indicate that somebody is exploiting an unpatched or zero-day vulnerability or misconfiguration in your cluster(s) early on in an attack (while you can still stop it before they do too much damage)

Summary or TL;DR

Identity and Access Management (IAM and Authx):

• In both EKS and GKE authentication (by default) is via their respective Identity and Access Management (IAM) services/identities.
• In AWS EKS authorization is handled entirely through its proprietary EKS Access Entries API — whereas in GCP it is controlled by GCP IAM and K8s RBAC (so you need to manage/monitor both).

Network Firewalling

• In AWS you have public vs. private subnets (based on whether the default route for the subnet is an Internet Gateway (IGW) or a NAT Gateway) whereas, in GCP, you can make any resource public by putting a public IP on it.
  - I know NAT isn’t a firewall — but it is often used as a rudimentary security layer around many AWS/EKS environments.
• In AWS, your firewall options are Network Access Control Lists (NACLs) assigned to subnets and Security Groups assigned to resources whereas, in GCP, they have their Cloud Next Generation Firewall (NGFW) at the VPC-level.
  - AWS also offers the optional AWS Network Firewall service which can be configured/managed in the cloud out-of-band from K8s
• In AWS, Security Group rules can reference other Security Group IDs for dynamic rules not based on (ever changing) IPs and fixed/inflexible CIDRs (e.g. you allow the application’s AWS SG ID ingress access on the database’s AWS SG) whereas, in GCP, you put tags on your resources and can reference those to allow for similarly dynamic/micro-segmented firewalling.
• AWS offers Security Group for Pods but GKE doesn’t offer tags for Pods
  - Tags in GKE can only secure Nodes not Pods — though you technically can configure Pods to NAT their egress traffic through Nodes if you need to use them with Pods
• Both EKS and GKE support K8s standard NetworkPolicies
• GKE supports two other types of NetworkPolicies:
  - FQDNNetworkPolicy — GKE-only and egress-only policies that let you you specify a DNS FQDN for your destination — and only available in the more expensive GKE Enterprise
  - CiliumClusterWideNetworkPolicies (but not the Namspaced CiliumNetworkPolicies strangely) — as Dataplane V2 is based on Cilium GKE makes their cluster-wide policies optionally available too
  - They block your ability to use the Layer 7 features of the CiliumClusterWideNetworkPolicies network policy though!

Configuration/Posture management

• GKE Enterprise has Kubernetes configuration/posture management but AWS doesn’t (either in EKS or via its services like Config)

Vulnerability/Supply-chain management

• Both EKS and GKE ensure that your Kubernetes control-pane is patched and provide tooling to help you keep them up-to-date.
• Both EKS and GKE both offer regularly patched Machine Images for their Nodes and tooling to help you keep them up-to-date
• AWS only offers container image scanning in your pipelines and/or registry (via Inspector outside of EKS) while GCP does it within GKE and with a runtime container image scanning capability (i.e. tells you which running workloads are using that image with those vulnerabilities)
• Both AWS and GCP offer the ability to cryptographically verify that your container images are the ones you built — AWS with AWS Signer and GCP with Binary Authorization.
  - These two appear to be incompatible with one another so implementing them would silo images to only be trusted/runable in one cloud or the other though

Runtime threat detection

• AWS has GuardDuty and GCP has Security Command Center to provide runtime threat detection for your EKS/GKE clusters
• AWS GuardDuty provides runtime threat detection against both the K8s API as well as the system calls on each Node — while GCP SCC appears to just provide the later.

Identity and Access Management (IAM)

One of the key areas of security is ensuring that only the right people, services or pipelines can only do the things they need to do on your cluster(s) — the principle of least privilege.

Authentication (authn)

Authentication is about ensuring you are who/what you say that you are. One area where EKS and GKE are the same is that they default to you authenticating with their underlying cloud’s IAM’s Users/Roles/Service Accounts. In both clouds there is a command in their respective CLIs to wire-up your cloud identity to a Kubernetes one for use with kubectl— in AWS it isaws eks update-kubecofig and in GCP it isgcloud container clusters get-credentials.

NOTE: In the case of both EKS and GKE you’ll need some permissions to their cloud managed K8s APIs (The EKS/GKE APIs as opposed to K8s one) in order to just run those CLI commands. In AWS that minimum access is eks:DescribeCluster while in GKE that it is the Role Kubernetes Engine Cluster Viewer.

Authorization (authz)

They differ a bit, though, when it comes to Authorization — which is about making sure that a given identity (now that we’re sure they are who/what they say they are from authentication) can only do what it needs to do. In Kubernetes this is usually managed through Kubernetes RoleBindings and ClusterRoleBindings. These maps an identity, such as from an Kubernetes built-in Service account or external OIDC provider, to an Role or ClusterRole defined in the cluster — which, in turn, defined the ‘rules’ of what that identity can do within the cluster and on what resources.

The distinction between RoleBindings and ClusterRole bindings is about Kubernetes Namespaces — which are a logical division within the cluster (similar to how an AWS Account or GCP Project is a logical division within the underlying cloud). While you can do some granular things within a Namespace you’ll find that with Kubernetes RBAC you often need to separate teams/tenants into different Namespaces, and limit them to Roles in those Namespaces (as opposed to ClusterRoles), to truly prevent one from accessing/interfering with another. This sharing of one cluster across multiple tenants is called multi-tenancy — and it is covered in the K8s documentation here.

In the case of EKS’s interaction with AWS IAM, though, these bindings are only managed through an AWS proprietary API called Access Entries these days (as opposed to directly against the K8s API). Note that this replaces a now-deprecated approach of defining the mappings in a ConfigMap called aws-auth. The reason for this is so that you can control what AWS identities have access to the cluster entirely through AWS APIs (including via CloudFormation or Terraform alongside setting up the cluster and all the other associated cloud infrastructure). It also makes this AWS API the sole “source of truth” for what AWS IAM identities have authorization to this EKS cluster — so you have only one place to look/manage.

Whereas, in GKE, you have the choice/combination of both cloud their native IAM as well as K8s RBAC to manage access to their K8s API. At the GCP IAM level you have these fairly blunt Roles you can assign an identity:

You’ll note that these are blunt in two ways — by default they are not granular as to which clusters in a given GCP Project an identity has access to. Though that granularity is possible if you assign tags to cluster(s) and then use tag-based IAM conditions on the role assignment. Even then, though, they also don’t let you specify that a particular identity only has access to a particular Namespace within a cluster either — just the whole thing. That’s why Google lets you also use the built-in Kubernetes RBAC in combination with them.

The way it would work is that you give an identity in GCP IAM just Kubernetes Engine Cluster Viewer (so they can authenticate to the clusters with gcloud container clusters get-credentials). And then you can give them either a RoleBinding or ClusterRoleBinding with the following convention to authorize them within a given Namespace and Cluster:

Here is an example K8s RoleBinding with each of those types reflected in it:

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: pod-reader-binding
  namespace: accounting
subjects:
# Google Cloud user account
- kind: User
  name: janedoe@example.com
# Kubernetes service account
- kind: ServiceAccount
  name: johndoe
# IAM service account
- kind: User
  name: test-account@test-project.iam.gserviceaccount.com
# Google Group
- kind: Group
  name: accounting-group@example.com
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

With this unique combination of K8s RBAC and GCP IAM that GKE offers, it can get difficult to get a full picture of who has access to a cluster. The popular open-source tool rbac-lookup can help with that. With a simple command like rbac-lookup rob --gke, it will show all the relevant K8s RBAC roles, along with all the relevant GCP IAM roles associated with GKE access for the matching user(s).

Have GKE manage EKS authX?

There is also a bit of a wildcard here — you can also ‘attach’ your EKS cluster to GKE Enterprise (formerly called Anthos). This, among other things, lets you manage authX for the connected EKS clusters via the same mechanisms as your GKE clusters (via their Connect API K8s API proxy). And it also lets you use the (arguably better) GKE console to manage all the clusters across both clouds as well. Especially if you use Google Workplace and so have standardized on Google identities as the main ones for your employees it can be tempting to follow this path to simplify/standardize everything around those and Google.

The thing worth noting here is that GKE (though, admittedly, in their more expensive ‘Enterprise’ mode) can manage EKS — but EKS can’t manage GKE in the same way. You could argue that EKS Hybrid Nodes can run on GCP — but that is a much less managed scenario than using the native ‘managed K8s’ on each cloud GKE gives you.

Network Firewalling

When it comes to controlling/firewalling network traffic on EKS or GKE there are two layers that you need to think about — the underlying cloud and the Kubernetes.

Network Firewall differences between GCP and AWS

• In AWS you have public vs. private subnets (based on whether the default route for the subnet is an Internet Gateway (IGW) or a NAT Gateway) whereas, in GCP, you can make any resource public by putting a public IP on it.
  - I know NAT isn’t a firewall — but it is often used as a rudimentary security layer around many AWS/EKS environments.
• In AWS, your firewall options are Network Access Control Lists (NACLs) assigned to subnets and Security Groups assigned to resources whereas, in GCP, they have their Cloud Next Generation Firewall (NGFW) at the VPC-level.
  - AWS also offers the optional AWS Network Firewall service which can be configured/managed in the cloud out-of-band from K8s
• In AWS, Security Group rules can reference other Security Group IDs for dynamic rules not based on (ever changing) IPs and fixed/inflexible CIDRs (e.g. you allow the application’s AWS SG ID ingress access on the database’s AWS SG) whereas, in GCP, you put can use either tags or service account identities associated with your VMs in firewall rules to allow for similar micro-segmentation.
• AWS offers Security Group for Pods with their CNI for EKS but GKE doesn’t offer the ability to use either tags or service accounts to be used for Pods in the same way
  - Tags in GKE can only secure Nodes not Pods — though you technically can configure Pods to NAT their egress traffic through Nodes if you need to use them with Pods
  - Similarly Service Accounts (via Workload Identity Federation) can’t be used for firewall rules in GKE either
  - Note that AWS SG for Pods is not enabled by default — the instructions linked above walk through enabling them
• Both EKS and GKE support K8s standard NetworkPolicies
  - EKS doesn’t do enforce them out-of-the-box. You can enable with the AWS CNI by following these instructions
  - Whereas GKE does enable/enforce these by default if you specify them in Autopilot clusters as well as those using their Dataplane V2 (which is what everybody should be using for new cluters these days)
• GKE supports two other types of NetworkPolicies:
  - FQDNNetworkPolicy — GKE-only and egress-only policies that let you you specify a DNS FQDN for your destination (note that these require GKE Enterprise)
  - CiliumClusterWideNetworkPolicies (but not CiliumNetworkPolicies strangely — seemingly because they already had FQDNNetworkPolicies?) — as Dataplane V2 is based on Cilium GKE makes their cluster-wide policies optionally
  - There is a bit BUT to CiliumClusterWideNetworkPolicies that I was surprised by though — when you try to specify the Layer 7 rules such as ones based on DNS FQDNs it is blocked by an admission controller as follows. It is very strange (and frustrating) that these policies appear to be there by that GCP blocks major functionality in them. I hope that this isn’t to drive you to their GKE Enteprise in order to get their proprietary FQDNNetworkPolicy and has a good technical reason — but regardless of why I found this surprising and very disappointing.

Error from server (GKE Warden constraints violations): error when creating "ccnp.yaml": admission webhook "warden-validating.common-webhooks.networking.gke.io" denied the request: GKE Warden rejected the request because it violates one or more constraints.
Violations details: {"[denied by gke-cilium-network-policy-limitation]":["L7 rules are not allowed in CiliumClusterwideNetworkPolicy"]}Requested by user: 'user@example.com', groups: 'system:authenticated'.

Configuration / Posture Management

How you configure EKS/GKE at the cloud API layer, at the K8s API layer, as well as the Specs of each workload running on it can impact the security of your cluster(s) and workloads. This is a major concern with Kubernetes because it allows you to ask for a quite insecure posture out-of-the-box for your workload(s) to enable it to schedule critical cluster infrastructure components/add-ons (such as the network driver (CNI), storage driver (CSI), observability tools, security tools, etc.) via Kubernetes as privileged DaemonSets. So, you need to take care to allow those things that need this access but to block those things that don’t.

A good starting point that is now built-in to Kubernetes (and therefore both EKS and GKE) is Pod Security Standards — which it can enforce via the Pod Security Admission Controller. These can help you to block the most common dangerous/insecure settings people can ask for in their Pods specs that would allow them to escape their containers and control the underlying Node and other Pods/containers that are running there.

It is worth noting that both EKS Auto Mode and GKE Autopilot are configured to block most container escapes. This is because the cloud provider has responsibility for these Nodes in this model and so harden and defend them more in line with that increased responsibility.

AWS doesn’t have an offering in this configuration/posture management space as part of their managed offerings (either EKS or their Config etc.) Whereas GCP offers a full posture management offering as part of GKE Enterprise called Policy Controller. This can even be used to secure EKS in addition to GKE as well if you attach the clusters to GKE Enterprise. Note that GKE Enterprise has additional costs associated with it.

GKE Security Posture showing a workload with a poor security posture (that undermines that of the whole cluster since it allows for container escape)

Vulnerability / Supply-Chain Management

There are a few main things that most organizations will want to scan for vulnerabilities and patch when there are critical ones:

• The Kubernetes Control Plane
• Their Node OS
• Their container images (including the base image/layers as well as all the open-source packages they’ve included from npm/pip/maven etc.)

Kubernetes Control Plane Vulnerabilities

For patches to supported versions of the Kubernetes control plane (e.g. 1.32), EKS calls these platform versions and GKE calls them patch releases. Both clouds offer these patches in a timely manner and help provide seamless upgrades to them as part of their managed service.

I’ll discuss the differences in the automated Control Plane upgrade process between EKS and GKE in the next blog post.

Node (OS/Machine Image) Vulnerabilities

Assuming that you’re using their provided machine images, both providers offer regularly patched versions of these images which you usually use to replace Nodes immutably rather than try to patch them in-place with both EKS and GKE — and a way to trigger updates of them as and when required.

They both also offer ‘minimal’ base images optimized for use as Kubernetes Nodes— Bottlerocket in the case of EKS and Container Optimized OS in the case of GKE — that have are hardened and with a much smaller attack surface. In the case of EKS, this is not the default (instead opting for one based on Amazon Linux) but Container Optimized OS is the default for GKE.

While neither EKS or GKE will tell you about known CVEs on your running Nodes (to let you know you should prioritize upgrading them to the latest machine image or patch them), their wider platforms do — via Amazon Inspector and GCP’s VM Manager. Note that they charge for both of these services — here is AWS and GCP pricing for these services (both of which charge for this based on the number of Instances/VMs that you’re scanning).

In my experience, if you ensure you upgrade your Nodes to the latest machine image regularly (every few weeks) — and especially if you use Bottlerocket or Container Optimized OS (which have a much smaller surface of potentially vulnerable packages/vulnerabilities) — that is usually enough to stay on top of any vulnerabilities at this layer.

I’ll discuss the differences in the automated Node upgrade process between EKS and GKE in the next blog post.

Container Image Vulnerabilities

AWS EKS itself doesn’t have a component to manage container image vulnerabilities. The main offering from AWS is part of its Inspector service — which can scan for vulnerabilities both in their Elastic Container Registry (ECR) as well as for you to trigger in your pipelines as a step/gate to getting the image pushed into one.

Example scan of a container image in the probe-test-app ECR repository

There are a few issues with this approach:

• You don’t know if a given image tag in ECR is actually/still running anymore (e.g. maybe you’ve recently updated it and fixed the vuln in a new tag and this is the old unfixed tag still in the registry)
• You don’t know (short of a good naming convention or tagging) which services/tenants are running the image
• You need to ensure that every image running in your cluster comes from your ECR and thus gets scanned

AWS Inspector pricing for ECR scanning is here — based on the number of images you’re scanning (with a discount for re-scans).

GKE has the better approach of actually scanning the images that are running in your clusters. This means that you don’t need to do that second step of trying to cross-reference the images to what’s running — as it shows you that X deployment in Y namespace in Z cluster running right now has these vulnerabilities.

Note that this requires GKE Enterprise and the pricing for that is here. Unlike Inspector this is charged based on the number of vCPUs of your clusters rather than the number of images getting scanned. It is more expensive than Inspector but it also includes not just this feature but many others too. So, whether it is worth it will depend on how many GKE Enterprise features you use beyond this one.

Container Supply-Chain Management/Verification

It is common to limit the containers that can run to those in your own private registries (which is arguably part of your configuration/posture management — but I’m considering it more as supply-chain management here). This ensures that:

• If they are your apps that they are the ones that passed your pipelines and you intended to run
• If they are 3rd party apps that you have vetted them and perhaps they passed your pipelines that scanned them for known vulnerabilities and what open-source licenses they have etc. before pulling them in to your private registry

This limiting of images to your private registries can be done via an admission controller. One isn’t included in EKS — but popular CNCF projects that you can implement to do it are OPA Gatekeeper or Kyverno. In the case of GKE, they include a managed OPA Gatekeeper as part of GKE Enterprise's Policy Controller that can do it.

Both AWS and GCP also offer the ability to cryptographically verify that your container images are the ones you built — AWS with AWS Signer and GCP with Binary Authorization. These two appear to be incompatible with one another, though, so implementing them would silo images to only be trusted/runable in one cloud or the other.

Runtime Threat Detection

The final line of defense with Kubernetes, as with most other cloud services, is Runtime Threat Detection. This is where a tool watches everything from the system calls on each Instance/VM, to those against your K8s and cloud provider’s APIs, to the DNS calls and network traffic etc. to determine if suspicious activity is happening that indicates an attack is underway. It is still possible, even if you have done all the right preventative things (configured your posture securely and have patched any known vulnerabilities etc.), that a zero-day vulnerability could be exploited in an attack — and a tool like this is the only way you’d pick that up.

In the case of AWS, their runtime threat detection tool is AWS GuardDuty. It can watch API calls to your K8s API using the GuardDuty EKS Protection feature. And it can watch all the system calls on your Kubernetes Nodes (covering all of your Pods/workloads) via an agent using GuardDuty Runtime Monitoring. You can see the sorts of findings that this would give you in their documentation here.

In the case of GCP, they have a runtime threat detection capability as part of their Security Command Center (SCC) service. While they don’t call out a capability that monitors your K8s API/audit trail for suspicious activity, they do have an agent-based approach for runtime threat of your workloads called Container Threat Detection. You can see the sorts of detections this would give you in their documentation here.

Note that for both of these options (GuardDuty and SCC) there are additional (and somewhat complex to predict) costs involved with enabling them — and you should do your research there before doing so.

Conclusion

I hope that was a good high-level introduction to how AWS EKS and GCP GKE handle security — and the differences between the two.

For my next, and perhaps final, blog post I’ll be looking at operations — with a focus particularly on their differences in their automation around patching/upgrading Nodes and Clusters to new Machine Images and versions of Kubernetes.