1 SCIENCE PASSION TECHNOLOGY Introduction to Scientific Writing 03 Experiments & Reproducibility Matthias Boehm Graz University of Technology, Austria Institute of Interactive Systems and Data Science Last update: Nov 11, 2021 2 706.015 Introduction to Scientific Writing – 03 Experiments & Reproducibility Matthias Boehm, Graz University of Technology, WS 2021/22 Announcements/Org  #1 Virtual Lectures Scientific Writing – 03 Experiments & Reproducibility Matthias Boehm, Graz University of Technology, WS 2021/22 Agenda  Experiments and Result Presentation  Reproducibility and RDM  Reminder: Paper Project

Outputs Distributing Workloads 2. Reproducibility Data Versioning Reproducibility For Developers Reproducibility For Developers For the Team Reproducibility For Developers For the Team For Production Back Models 3. Clarity / Organizational Trust 4. Automation (CI/CD) Summary 1. Autonomy 2. Reproducibility 3. Data Provenance 4. Automation Demo Demo Demo Demo Demo Contact Me Twitter: @samkreter

communication through scientific writing  Best practices for effective scientific reading, writing, and reproducibility  Grading  Overall: pass/fail (no detailed 1-5 grades)  Includes submitted paper and final 15pm, optional]  02 Scientific Reading and Writing [Nov 04, 6pm, optional]  03 Experiments, Reproducibility, and Projects [Nov 11, 6pm, optional] ...  04 Project Presentations [Jan 13, 6pm, mandatory] parameters Structure of Scientific Papers 02 Scientific Reading and Writing 03 Experiments and Reproducibility 23 706.015 Introduction to Scientific Writing – 01 Introduction and Overview Matthias Boehm

access safety Thread Safety Arithmetic safety Definition safetySoftware Supply Chain Safety ? Reproducibility Traceability Release delivery timeYOU DON'T NEED TOAchieve safety in C++ todayAchieve safety compile) 7 8 set_property(TARGET atarget 9 PROPERTY JOB_POOL_LINK link) 10 11 12 13 14 15Reproducibility Given the same inputs and configuration, a target should yield identical outputsReproducibility outputsReproducibility Level 1 - Make, Ninja, FASTBuild Provide no real facility to ensure reproducibility and, thus, hermeticity is something that the developers have to take care ofReproducibility Level 2 - Meson

propagation (the bane of all freshman physics labs) 24Qualities of experiments 1. Precision 2. Reproducibility 3. Accuracy 25Precision Precision maximizes information content Use a test framework with right place with the right context (no red herrings!) Limit tests to the code in question 26Reproducibility (handling nondeterministic results) Most scientific experiments have measurement error or noise Solutions: Isolate Subtract Detect and eliminate erroneous results Statistical analysis 27Reproducibility (handling unreliable results) Unreliable or "flaky" tests: depend on timing or external state

signifi- cant time for developers and Continuous Integration servers, while also improving the reproducibility and traceability of artifacts. A package is defined by a “conanfile.py”. This is a file that with version ranges, the concept of revisions and a brief introduction to lockfiles to achieve reproducibility of the dependency graph. 4.1.1 Build a simple CMake project using Conan Let’s get started with For those situations, Conan uses automatic revisions to be able to provide traceability and reproducibility of those changes. Lockfiles are a common mechanism in package managers to be able to reproduce

signifi- cant time for developers and Continuous Integration servers, while also improving the reproducibility and traceability of artifacts. A package is defined by a “conanfile.py”. This is a file that with version ranges, the concept of revisions and a brief introduction to lockfiles to achieve reproducibility of the dependency graph. 4.1.1 Build a simple CMake project using Conan Let’s get started with For those situations, Conan uses automatic revisions to be able to provide traceability and reproducibility of those changes. Lockfiles are a common mechanism in package managers to be able to reproduce

signifi- cant time for developers and Continuous Integration servers, while also improving the reproducibility and traceability of artifacts. A package is defined by a “conanfile.py”. This is a file that version ranges to update easily to the latest one within the range, and lockfiles to achieve reproducibility. See also: • Read the conan config install-pkg command reference 2.15. Package lists 13 Conan with version ranges, the concept of revisions and a brief introduction to lockfiles to achieve reproducibility of the dependency graph. 4.1.1 Build a simple CMake project using Conan Let’s get started with

signifi- cant time for developers and Continuous Integration servers, while also improving the reproducibility and traceability of artifacts. A package is defined by a “conanfile.py”. This is a file that version ranges to update easily to the latest one within the range, and lockfiles to achieve reproducibility. See also: • Read the conan config install-pkg command reference 2.15. Package lists 13 Conan with version ranges, the concept of revisions and a brief introduction to lockfiles to achieve reproducibility of the dependency graph. 4.1.1 Build a simple CMake project using Conan Let’s get started with

signifi- cant time for developers and Continuous Integration servers, while also improving the reproducibility and traceability of artifacts. A package is defined by a “conanfile.py”. This is a file that version ranges to update easily to the latest one within the range, and lockfiles to achieve reproducibility. See also: • Read the conan config install-pkg command reference 2.15. Package lists 13 Conan with version ranges, the concept of revisions and a brief introduction to lockfiles to achieve reproducibility of the dependency graph. 4.1.1 Build a simple CMake project using Conan Let’s get started with