release-1.15 (continued from previous page) schema = (Schema.new_builder() .column('id', DataTypes.TINYINT()) .column('data', DataTypes.STRING()) .build()) table = table_env.from_descriptor( TableDescriptor BYTE(), Types.STRING()])) table = t_env.from_data_stream(ds, Schema.new_builder() .column("id", DataTypes.TINYINT()) .column("data", DataTypes.STRING()) .build()) table.get_schema() [8]: root |-- id: TINYINT selecting a column does not trigger the computation but it returns a Column Expression instance. [15]: from pyflink.table.expressions import col type(table.id)==type(col('id')) [15]: True These Column Expressions

release-1.16 (continued from previous page) schema = (Schema.new_builder() .column('id', DataTypes.TINYINT()) .column('data', DataTypes.STRING()) .build()) table = table_env.from_descriptor( TableDescriptor BYTE(), Types.STRING()])) table = t_env.from_data_stream(ds, Schema.new_builder() .column("id", DataTypes.TINYINT()) .column("data", DataTypes.STRING()) .build()) table.get_schema() [8]: root |-- id: TINYINT selecting a column does not trigger the computation but it returns a Column Expression instance. [15]: from pyflink.table.expressions import col type(table.id)==type(col('id')) [15]: True These Column Expressions

total fare collected Lab 1 -- Ride Cleansing Transforming Data Transforming Data public static class EnrichedRide extends TaxiRide { public int startCell; public int endCell; public filter(new RideCleansing.NYCFilter()) .map(new Enrichment()); enrichedNYCRides.print(); public static class Enrichment implements MapFunction { @Override public EnrichedRide taxiRide) throws Exception { return new EnrichedRide(taxiRide); } } FlatMap Function public static class NYCEnrichment implements FlatMapFunction { @Override public void

main steps to develop a Spark stuctured streaming: ▶ 1. Defines a query on the input table, as a static table. • Spark automatically converts this batch-like query to a streaming execution plan. ▶ 2 main steps to develop a Spark stuctured streaming: ▶ 1. Defines a query on the input table, as a static table. • Spark automatically converts this batch-like query to a streaming execution plan. ▶ 2 main steps to develop a Spark stuctured streaming: ▶ 1. Defines a query on the input table, as a static table. • Spark automatically converts this batch-like query to a streaming execution plan. ▶ 2

flatMap(new MatchFunction());
 Java example 20 Vasiliki Kalavri | Boston University 2020 public static class EnrichmentFunction extends RichCoFlatMapFunction> the job is started or in the case of a failure. Vasiliki Kalavri | Boston University 2020 public static class CounterSource extends RichParallelSourceFunction implements ListCheckpointed {

high-rate append-only updates Data Warehouse • complex, offline analysis • large and relatively static and historical data • batched updates during downtimes, e.g. every night Streaming Data Warehouse pre-aggregated, pre-processed streams and historical data Data Management Approaches 4 storage analytics static data streaming data Vasiliki Kalavri | Boston University 2020 DBMS vs. DSMS DBMS DSMS Data persistent

transactions, likes, comments • Analytics on user activity • Filtering, aggregation, joins with static data (e.g. user profile data) Examples • online A/B testing • trending topics • sentiment analysis

easy to measure, estimates can be computed in a straight-forward manner. • Estimations based on static operator selectivities and heuristics are unsuitable for frequent load fluctuations. • Naive

by the metaspace, the size of which is unlimited by default and holds class metadata as well as static content. There is a JIRA Ticket6 to limit the size to 250 megabyte by default • The biggest driver

of B < 0.5 Operator re-ordering B A A B ??? Vasiliki Kalavri | Boston University 2020 17 • A static graph transformation that enables re-ordering at runtime • It dynamically routes data after