o Istiod MEM bumped with large numbers of Knative Services (#25532) Mem usage optimization of pilot resolved this issue. • Tune CPU/MEM to ensure enough capacity Leveraged Metrics to monitor Istio Istiod. o From envoy logs, transient 503 UH "no healthy upstream" errors. o From Grafana dashboard, Pilot Pushes shows long latencies. • Detect and analyze Istio scalability issue #IstioCon o Radom peaks Knative Service provisioning o PILOT_ENABLE_EDS_DEBOUNCE=true,PILOT_DEBOUNCE_AFTER=100ms and PILOT_DEBOUNCE_MAX=10s are the env vars on pilot that can be tuned. o Set PILOT_DEBOUNCE_AFTER=1s helps under

sidecar Envoys ○ Measure Config convergence time ■ Time taken by all sidecars to get config from Pilot without any errors ■ For thousands of services & endpoints ■ With different churn rates of Pods time from single Pilot instance to 0 - 3,000 sidecars < 1 second ○ Pilot CPU & memory within acceptable limits: < 10 cores, 25 GB memory ○ Pilot can scale horizontally ● Need to tune PILOT_DEBOUNCE_AFTER PILOT_DEBOUNCE_AFTER, PILOT_DEBOUNCE_MAX, PILOT_PUSH_THROTTLE, etc. params of Istio Pilot #IstioCon Future Direction ● Support for on-demand config pushes to Envoy via Incremental XDS ● Support for multiple trust

services (Pilot, Mixer, CA) accessible from the VMs ○ (optional) Kubernetes DNS server accessible from the VMs ● Onboard steps ○ Setup Internal Load Balancers (ILBs) for Kube DNS, Pilot, Mixer and

and can view the resulting calculations for the loan-repayment amount or the interest rate. • Data Pilot: Enables you to compare, combine and arrange large amounts of data. It helps you pull in raw data databases, cross-tabulate, summarise and convert the data into meaningful information. You can use Data Pilot to create interactive tables, which allows the data to be frequently arranged, rearranged or summarised