Database Table Partition Spec Manifest File TableMetadata Snapshot Current Table Version Pointer Apac2e Ice-er1 Bas3c Part3t354- f f3 Part3t354-2 f4 f5 Part3t354-3 Ma43fest- Ma43fest-2

stream into m = 2p = 4 sub-streams. Consider the input elements {5, 14, 5, 2, 8, 1, …} Substream Address Counter S0 00 S1 01 S2 10 S3 11 ??? Vasiliki Kalavri | Boston University 2020 11 Stochastic stream into m = 2p = 4 sub-streams. Consider the input elements {5, 14, 5, 2, 8, 1, …} Substream Address Counter S0 00 S1 01 S2 10 S3 11 • x1=5, h5(5) = 00101 • x2=14, h5(14) = 10110 • x3=5 stream into m = 2p = 4 sub-streams. Consider the input elements {5, 14, 5, 2, 8, 1, …} Substream Address Counter S0 00 S1 01 S2 10 S3 11 • x1=5, h5(5) = 00101 • x2=14, h5(14) = 10110 • x3=5

What data structure would you use to: • Filter out all emails that are sent from a suspected spam address? • Filter out all URLs that contain malware? • Filter out all compromised passwords? • Remove What data structure would you use to: • Filter out all emails that are sent from a suspected spam address? • Filter out all URLs that contain malware? • Filter out all compromised passwords? • Remove

Vasiliki Kalavri | Boston University 2020 Exactly-once in Google Cloud Dataflow Checkpointing to address non-determinism • Each output is checkpointed together with its unique ID to stable storage before

operators according to the number of available cores / threads • Fused operators can share the address space but use separate threads of control • avoid communication cost without losing pipeline

cause imbalance w2 w1 w3 ??? Vasiliki Kalavri | Boston University 2020 Addressing skew • To address skew, the system needs to track the frequencies of the partitioning key values. • We can then

run --python examples/python/table/word_count.py --pyFiles file:///user. ˓→txt,hdfs:///$namenode_address/username.txt For example, if you have a directory named myDir which has the following hierarchy:

run --python examples/python/table/word_count.py --pyFiles file:///user. ˓→txt,hdfs:///$namenode_address/username.txt For example, if you have a directory named myDir which has the following hierarchy: