- keys in sorted array - giving O(logn) access Speaker notesJOB SYSTEM • Schedules jobs on many worker threads • Uses a Counter to synchronise between Jobs ▪ Leading "wait" counter marks job as runnable when finished • Inspired by Naughty Dog and CD Projekt RED job systems 12Jobs run on worker threads Number worker threads limited to logical processors Counters used to sync between jobs Each job uses std::vector owned_nodes_; std::vector intersecting_nodes_; }; 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 24GridCell stores sorted arrays of GridNodes • Array of owned nodes - cell contains

with their own logic and, if needed, data, queue, etc) ○ A Signal Tree (which has as many leaf nodes as there are work contracts in the group) ● Threads are brought to the “task” rather than the “task” throughput than the average MPMC queue at scale ○ Approximately 1/2N memory requirement (N = number of nodes) Alternative: Work Contracts Work Contract Group Work Contract (Logic) Work Contract Work Contract with their own logic and, if needed, data, queue, etc) ○ A Signal Tree (which has as many leaf nodes as there are work contracts in the group) ● Threads are brought to the “task” rather than the “task”

detect and handle recoverable errorsWhat is “parallel”? • Use multiple hardware resources – Nodes, cores, SIMD, … • To accomplish >1 work item at the same time • To improve performance – Latency execution; handling constrains order • Errors could lead to deadlock (waiting forever) – e.g., 1 worker drops out before collective synchronization • Correct handling requires communication – Data movement

frames on the CPU timeline. 4445 COROUTINE JOB SYSTEM isa::await_suspend Scheduler Worker Worker Worker Worker Job Queue Job Queue Job Queue Job coroutine_handle Job::Job *isa == initial_suspend_awaitable final_suspend And voila! This is our system. 4546 isa::await_suspend Scheduler Worker Worker Worker Worker Job Queue Job Queue Job Queue Job coroutine_handle Job::Job *isa == initial_suspend_awaitable glues our system and coroutine semantics together. 4647 isa::await_suspend Scheduler Worker Worker Worker Worker Job Queue Job Queue Job Queue Job Job::Job *isa == initial_suspend_awaitable get_return_object

C B HPQ LPQ Worker 1 GPU stream Worker 2 Block Poll Enqueue Steal Offload Callback Polling Wait Query • Assume two CPU threads (workers) and one GPU stream • Each worker has: High-priority the same worker as soon as possible to avoid another worker stealing the task 60Taro’s Scheduler Taro: https://github.com/dian-lun-lin/taro D A C B HPQ LPQ A Worker 1 GPU stream Worker 2 Block Offload Callback Polling Wait Query • Assume two CPU threads (workers) and one GPU stream • Each worker has: High-priority queue (HPQ): store suspended tasks Low-priority queue (LPQ): store

know what it solves and how to defend against edge cases) Thread-Stealing Work Queue worker worker worker Work Engine Foo Subsystem A Bar Subsystem B BazCharley Bay - charleyb123 at gmail dot as special handling to execute one work item many times) Thread-Stealing Work Queue worker worker worker Work Engine Foo Subsystem A Bar Subsystem B BazCharley Bay - charleyb123 at gmail dot work progress (but indirect monitoring can be implemented) Thread-Stealing Work Queue worker worker worker Work Engine Foo Subsystem A Bar Subsystem B BazCharley Bay - charleyb123 at gmail dot

monitoring code ● Let’s try to add more worker threads to thisComplexity level 2 : Multiple worker threadsComplexity level 2 : Multiple worker threads ● Worker threads cannot call the callback handlers the worker threads and calls the callbacks? ■ Inefficient! Do you know why? ○ Main thread releases GIL before blocking, workers acquire GIL before callbackComplexity level 2 : Multiple worker threadsComplexity threadsComplexity level 2 : Multiple worker threads ● Seems scalable ● C++ dispatching thread doesn’t even need GIL ○ Python is not inherently blocked!Complexity level 3 : Python background tasksComplexity

on_scheduler_entry(bool is_worker) override { if (is_worker) { int tid = tbb::this_task_arena::current_thread_index(); tracy::SetThreadName(std::string("tbb worker #").append(std::to_string(tid)) append(std::to_string(tid)).c_str()); } } void on_scheduler_exit(bool is_worker) override { } }; 50 Tips & Tricks Name your threads!example: don’t call memcpy() directly… call some my_memcpy() which (it’s basically a stackable clock/sustain plot!) 87 e.g.: task starts in main thread, moves between worker threads, and gets retired on a reclaimer thread Tips & Tricks Begin/End (and track) zone on different

on_scheduler_entry(bool is_worker) override { if (is_worker) { int tid = tbb::this_task_arena::current_thread_index(); tracy::SetThreadName(std::string("tbb worker #").append(std::to_string(tid)) append(std::to_string(tid)).c_str()); } } void on_scheduler_exit(bool is_worker) override { } }; 50 Tips & Tricks Name your threads!example: don’t call memcpy() directly… call some my_memcpy() which (it’s basically a stackable stairstep plot!) 63 e.g.: task starts in main thread, moves between worker threads, and gets retired on a reclaimer thread Tips & Tricks Begin/End (and track) zone on different

Variables (1/4) ● Perhaps a stranger thing when working with threads ○ Idea: ■ Given 2 threads (a worker and reporter), we want one thread to wait on the result of the other ■ Condition_variable allows Variables (2/4) ● Perhaps a stranger thing when working with threads ○ Idea: ■ Given 2 threads (a worker and reporter), we want one thread to wait on the result of the other ■ Condition_variable allows Variables (3/4) ● Perhaps a stranger thing when working with threads ○ Idea: ■ Given 2 threads (a worker and reporter), we want one thread to wait on the result of the other ■ Condition_variable allows