correctness ??? Vasiliki Kalavri | Boston University 2020 Automatic Scaling Control 4 ??? Vasiliki Kalavri | Boston University 2020 The automatic scaling problem 5 Given a logical dataflow with sources S2 π=2 π=3 logical dataflow physical dataflow ??? Vasiliki Kalavri | Boston University 2020 Automatic scaling overview 6 scaling controller detect symptoms decide whether to scale decide much to scale metrics policy scaling action ??? Vasiliki Kalavri | Boston University 2020 Automatic scaling requirements 7 ▸ Accuracy ▸ no over/under-provisioning ▸ Stability ▸ no oscillations

(historical) and stream processing 6. Ensure availability despite failures 7. Support distribution and automatic elasticity 8. Offer low-latency 7 2005 Vasiliki Kalavri | Boston University 2020 actions, typically exact • Challenges: computation progress, fault-tolerance and result guarantees, automatic scaling and state migration, out-of-order processing 37 Vasiliki Kalavri | Boston University 2020

JobManager cannot restart the application until enough slots become available. • Restart is automatic if there is a ResourceManager, e.g. in a YARN setup • A manual TaskManager re-start or a backup

Profitability: under what conditions does the optimization improve performance? • can the decision be automatic? • Safety: under what conditions does the optimization preserve correctness? • maintain state