Python Virtual Environment . . . . . . . . . . . . . . . . . . . 24 1.3.2.2 O2: How to add Python Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 1.3.3 JDK issues . . . . . . . . . different projects by creating a separate environment for each project. It is a directory tree which contains its own Python executable files and the installed Python packages. It is useful for local development 18 20:54 flink-dianfu-python-B-7174MD6R-1908. ˓→local.log Besides, you could also check if the files of the PyFlink package are consistent. It may happen that you have installed an old version of PyFlink

Python Virtual Environment . . . . . . . . . . . . . . . . . . . 24 1.3.2.2 O2: How to add Python Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 1.3.3 JDK issues . . . . . . . . . different projects by creating a separate environment for each project. It is a directory tree which contains its own Python executable files and the installed Python packages. It is useful for local development 18 20:54 flink-dianfu-python-B-7174MD6R-1908. ˓→local.log Besides, you could also check if the files of the PyFlink package are consistent. It may happen that you have installed an old version of PyFlink

delete fileO 写T思l 1N5oFitioA ,elete File和nR SeD3umP大Qi己SeD3um 的,ata FileS 04I3M 2N-DualitJ ,elete File和n RSeD3um小Qi己SeD3um 的,ata FileS 04I3O 读取思l *CClJ ,eletioA *CClJ ,eletioA 5AnFDeNOSRTVU :1 :2 :3 f4 Ice4erg/Are3m1riAer Ice4erg/Are3m1riAer Ice4erg/Are3m1riAer 1 1riAe records Ao D3A3/DeleAe Files. F量文E集I1A4ns4cCion提D /4ACiCion-2 -cebeAg .eC4sCoAe /4ACiCion-1 /4ACiCion-3 -cebeAg D4C4 )enCeA ((3-2 -cebeAgSCAe4m2AiCeA -cebeAgSCAe4m2AiCeA -cebeAgSCAe4m2AiCeA 1 2AiCe AecoAds Co D4C4/DeleCe Files. -cebeAgFiles)ommiCCeA 2 EmiC compleCed D4C4File Co commiCCeA. f1 f2 f3 f4 F量文E集I1A4ns4cCion提D

Assignment #3 contributes 20% 7 Vasiliki Kalavri | Boston University 2020 Grading Scheme (2) Final Project (50%): • A real-time monitoring and anomaly detection framework • To be implemented individually Deliverable Available Due Assignment 1 1/30 2/12 Assignment 2 2/13 2/26 Assignment 3 3/3 3/16 Final Project 3/17 4/30 2/18: No Class, Self-study 2/25: Last Day to DROP Clases (without a ‘W’ grade) 4/3: sure to check the Official Semester Dates 11 Vasiliki Kalavri | Boston University 2020 Final Project You will use Apache Flink and Kafka to build a real-time monitoring and anomaly detection framework

TCP socket connection. ssc.socketTextStream("localhost", 9999) ▶ File stream • Reads data from files. streamingContext.fileStream[KeyClass, ValueClass, InputFormatClass](dataDirectory) streamingContext TCP socket connection. ssc.socketTextStream("localhost", 9999) ▶ File stream • Reads data from files. streamingContext.fileStream[KeyClass, ValueClass, InputFormatClass](dataDirectory) streamingContext

Kalavri
 vkalavri@bu.edu Spring 2020 1/28: Stream ingestion and pub/sub systems Streaming sources Files, e.g. transaction logs Sockets IoT devices and sensors Databases and KV stores Message queues

in fully dynamic streams with fixed memory size. TKDD 2017. https://www.kdd.org/ kdd2016/papers/files/rfp0465-de-stefaniA.pdf Further reading

ETL process complex fast and light-weight ETL: Extract-Transform-Load
 e.g. unzipping compressed files, data cleaning and standardization 6 Vasiliki Kalavri | Boston University 2020 1. Process events

can be configured7. • Mapped memory is usually close to zero as Flink does not use memory-mapped files. In a containerized environment you should additionally monitor the overall memory consumption of