work to background threads Anthony Williams Just Software Solutions Ltd https://www.justsoftwareresolutions.co.uk September 2020 ## Get off my thread: Techniques for moving work to background threads to move the work, we need to prevent blocking on our event-handling threads. void event_handler() { auto handle = spawn_background_task(); handle.wait(); // no benefit } ## Aside: Non-Blocking vs off the current thread: ☑ Spawn a new thread for each event handler ☑ Pass data to a dedicated background thread ## How do we move work off the current thread? Possible ways to move the work off the current

std::thread library. We will introduce topics with pragmatic code examples introducing the ideas of threads and locks, and showing how these programming primitives enable concurrency. In addition, we will locks, conditional variables, promises/futures). Attendees will leave this session being able to use threads, locks, and start thinking about architecting multithreaded software. All materials and code samples /8/a2489eb71996fa476915c1f1c2383f3c/p35_1.jpg) Memory 16 GB GDDR6 SDRAM 512 MB DDR4 RAM (for background tasks) $ ^{[1]} $ Storage Custom 825 GB SSD Removable storage Internal (user upgradeable) NVMe

l# → background-parameters">Support setting the maximum limit on resource → usage for background tasks of resource control By setting a maximum percentage limit on background tasks of resource their resource consumption based on the → needs of different application systems. This keeps background task → consumption at a low level and ensures the quality of online services → . background tasks of resource control #56019 @glorv TiDB resource control can identify and lower the priority of background tasks. In certain scenarios, you might

l# → background-parameters">Support setting the maximum limit on resource → usage for background tasks of resource control By setting a maximum percentage limit on background tasks of resource their resource consumption based on the → needs of different application systems. This keeps background task → consumption at a low level and ensures the quality of online services → . background tasks of resource control #56019 @glorv TiDB resource control can identify and lower the priority of background tasks. In certain scenarios, you might

OVERVIEW 1. Background • What is data building? • Differences from Game Code • Assumptions and Concepts 2. Techniques • Keep Threads Busy • 3D Caching • Optimise Sorting • Avoid Blocking Threads 3. Questions Questions Speaker notes Three sections Background What data building is Differences from normal game code Concepts used in presentation Techniques I've used to optimise the data building system myself, not any employer Examples and code are my recreations from my memory and knowledge. ## BACKGROUND ## WHAT IS DATA BUILDING? • Called: compiling, converting, generating, “cooking”, or “baking”

number of concurrent threads during the Resolve Locks and Delete Range steps of the Garbage Collection (GC) process. Before v8.3.0, this variable only controls the number of threads during the Resolve Locks Collection) efficiency by supporting concurrent range deletion. You can control the number of concurrent threads using tidb_gc_ $ \hookrightarrow $ concurrency #54570 @ekexium - Improve the performance of bulk |配置|||||||||||| |DM WebUI|E|E|E|E|E|E|E|N|N|N|N| |Foreground Quota Limiter|Y|Y|Y|Y|Y|Y|E|N|N|N|N| |Background Quota Limiter|E|E|E|E|E|E|N|N|N|N|N| ![OCR图片](/uploads/documents/3/b/1/a/3b1a0439128acad53eceaf645a670e4f/p77_1

Installation ## What is a Task Queue? Task queues are used as a mechanism to distribute work across threads or machines. A task queue’s input is a unit of work called a task. Dedicated worker processes constantly Amazon SQS, (exp) • and more... • Concurrency • prefork (multiprocessing), • Eventlet, gevent • threads/single threaded • Result Stores • AMQP, Redis • memcached, MongoDB • SQLAlchemy, Django ORM celery[eventlet] for using the eventlet pool. celery[gevent] for using the gevent pool. celery[threads] for using the thread pool. ## Transports and Backends celery[librabbitmq] for using the librabbitmq

of users. - VLLM_CPU_OMP_THREADS_BIND: specify the CPU cores dedicated to the OpenMP threads. For example, VLLM_CPU_OMP_THREADS_BIND=0-31 means there will be 32 OpenMP threads bound on 0-31 CPU cores. VLLM_CPU_OMP_THREADS_BIND=0-31|32-63 means there will be 2 tensor parallel processes, 32 OpenMP threads of rank0 are bound on 0-31 CPU cores, and the OpenMP threads of rank1 are bound on 32-63 CPU cores and using CPU 0-29 for OpenMP: ```bash $ export VLLM_CPU_KVCACHE_SPACE=40 $ export VLLM_CPU_OMP_THREADS_BIND=0-29 $ vllm serve facebook/opt-125m ``` - If using vLLM CPU backend on a machine with hyper-threading

[http://eventlet.net/] or gevent [http://gevent.org/]. Tasks can execute asynchronously (in the background) or synchronously (wait until ready). Celery is used in production systems to process millions celery.task import task @task def add(x, y): return x + y You can execute the task in the background, or wait for it to finish: >>> result = add.delay(4, 4) >>> result.wait() # wait Starting/Stopping the RabbitMQ server To start the server: $ sudo rabbitmq-server you can also run it in the background by adding the -detached option (note: only one dash): $ sudo rabbitmq-server -detached Never use

If you want the best performance on CPU, it will be very critical to isolate CPU cores for OpenMP threads with other thread pools (like web-service event-loop), to avoid CPU oversubscription. - If using vllm-openai --tag vllm/vllm-openai #↵ →optionally specifies: --build-arg max_jobs=8 --build-arg nvcc_threads=2 Note: By default vLLM will build for all GPU types for widest distribution. If you are just building Number of threads to use for nvcc # By default this is 1. # If set, `MAX_JOBS` will be reduced to avoid oversubscribing the CPU. "NVCC_THREADS": lambda: os.getenv("NVCC_THREADS", None)