Lei Tang Google April 26, 2022 Service mesh security best practices: from implementation to verification Who are we? Anthony Roman Istio Github: anthony-roman Lei Tang Istio Github: lei-tang IDS Firewall User AuthN/Z Data Loss Prevention Certificate Authority K8s Network Policy K8s RBAC Audit Logging Image Verification Admission Control Workload Identity K8s RBAC K8s

issues found ● 5 system resource exhaustion ● 1 arbitrary file write ● 1 missing file close ● 1 certificate skipping ● 1 case unhandled errors ● 1 case of using a deprecated library ● 1 race condition them below: ● Certificate management ● Authentication ● Authorization ● Policy Enforcement Points (PEPs) ● A set of Envoy proxy extensions to manage telemetry and auditing Certificate management Alongside communicates with Istiod to automate key and certificate rotation, like so: Istio-agent has two functions: 1. To receive SDS requests from Envoy and send certificate signing requests to the CA which typically

communication using mTLS between all services • Configurable short-lived certificates • On the fly certificate renewals with no service downtime • Unified simplified configuration to enable mTLS for all services and certificate pairs • Private keys and certificates are stored in keystore and truststore files in JKS or PKCS12 or PEM format Challenges – Kafka broker SSL with client auth 5 • Certificate renewal Challenges – Certificate renewal 6 • Client certificates has be created for each separate client identity • Client certificates may take different formats (JKS, PEM, etc) • Client certificate renewal may

Gojek Agenda ● GoPay & Istio ● Before mutual TLS ● Implementing mutual TLS ○ Centralized Certificate Management ○ Ingress mutual TLS ○ Egress mutual TLS ● Challenge & Future Works GoPay & Istio used by all services) Implementing Mutual TLS Centralized Certificate Management ● Central certificate management manage our certificate lifecycle for HTTPS and mutual TLS communication. ● Renew AuthorizationPolicy to add IP allow listing Egress Mutual TLS ● Using Egress TLS origination ● Certificate is mounted in the client deployments using annotation sidecar.istio.io/userVolumeMount sidecar

was created which matched areas of code with specific security controls (e.g. service discovery, certificate lifecycle, side car injection) to focus testing efforts. Istio does not currently have a reference the only options included are how to “Harden Docker Container Images” and “Extending Self-Signed Certificate Lifetime”. There’s an op- portunity to highlight the impact of different securty options and expand attack surface that can be abused to extract the entire dynamic configuration of the proxy sans certificate private keys which could be abused in the event of a workload compromise or the exploitation of

VM's mesh identity (certificate) ■ based on a platform-specific identity ■ w/o a platform-specific identity ● using a short-lived K8s service account token ● Automatic certificate rotation ● Validation Alternative opts ○ Current: Fetch and exchange a k8s token for a bootstrap certificate, then place that bootstrap certificate on the VM ■ Dependency on K8s API server ■ Requires creating an RBAC impersonation Limitations to audit (proactively secure) ● VM cert extensibility ○ No support for workload certificate attributes #IstioCon Security & Usability Limitations (cont.) ● Access management: CNI needs

mutual TLS (mTLS) Option to encrypt intra-CNF traffic via mTLS Autonomous PKI service for certificate lifecycle management at scale What Do You Get From Istio? Traffic Management Powerful ● Integrate with PKI minted Intermediate CA ● Enable ECC certificates ● Configure workload certificate TTLs ● Enable strict mutual TLS (mTLS) instead of auto ● Use dedicated egress gateways Tuning rights reserved. ● Istio architectural changes ● SPIFFE only certificates ● Configuring workload certificate TTLs ● RSA to ECC migration ● Missing www-authenticate header ● Tuning per-workload proxy concurrency

As of Istio 1.7.7+, 1.8.2+ and 1.9.0+ there is no longer the restriction that a plugged in CA certificate must use ECC cryptography (using ECDSA P-256) to use this feature ● Only ECDSA P-256 is supported certificateChain.inlineBytes' | \ sed 's/"//g' | base64 --decode | openssl x509 -noout -text Certificate: Data: Version: 3 (0x2) Serial Number: … Signature Algorithm: ASN1 OID: prime256v1 NIST CURVE: P-256 istiod will generate a self-signed CA certificate using RSA if plugged in custom CA certificates aren’t specified #IstioCon MeshConfig support

Dataflow After Acceleration(same host) Istio Meetup China ebpf Background Knowledge Loader & Verification Architecture https://ebpf.io/what-is-ebpf/ Istio Meetup China ebpf Background Knowledge map

server side’s “envoy proxies” verify each other’s identities before sending requests. • If the verification is successful, then the client-side proxy encrypts the traffic, and sends it to the server- side