# Scrapy Documentation Release 2.7.1 Scrapy developers Nov 02, 2022 Powered by TCPDF (www.tcpdf.org) ## FIRST STEPS 1 Getting help 3 2 First steps 5 2.1 Scrapy at a glance 5 2.2 Installation guide 7 2.3 Scrapy Tutorial 11 2.4 Examples 23 3 Basic concepts 25 3.1 Command line tool 25 3.2 Spiders 35 3.3 Selectors 48 3.4 Items 66 3.5 Item Loaders 72 3.6 Scrapy shell 81 3 13 Jobs: pausing and resuming crawls 217 5.14 Coroutines 218 5.15 asyncio 222 6 Extending Scrapy 227 6.1 Architecture overview 227 6.2 Downloader Middleware 230 6.3 Spider Middleware 247

### Scrapy 2.7 documentation Scrapy is a fast high-level web crawling [https://en.wikipedia.org/wiki/Web_crawler] and web scraping [https://en.wikipedia.org/wiki/Web_scraping] framework, used to crawl the scrapy tag [https://stackoverflow.com/tags/scrapy]. • Ask or search questions in the Scrapy subreddit [https://www.reddit.com/r/scrapy/]. • Search for questions on the archives of the scrapy-users forum/scrapy-users]. • Ask a question in the #scrapy IRC channel. • Report bugs with Scrapy in our issue tracker [https://github.com/scrapy/scrapy/issues]. • Join the Discord community Scrapy Discord

API 9 2.4 New build system integrations 10 2.5 New custom user commands 10 2.6 New CLI 10 2.7 New deployers 10 2.8 New package_id 11 2.9 compatibility.py 11 2.10 New lockfiles 11 2.11 New used more easily for automation, especially in CI/CD systems. See also: • Commands reference ### 2.7 New deployers Conan 2 implements “deployers”, which can be called in the command line as conan install 8 or 2.0. Likewise requires = "pkg/[~2.5]" will include 2.5.0 and 2.5.3, but exclude 2.1, 2.7, 2.8. The caret ^ operator is very similar to the tilde, but allowing variability over the last defined

# Weblate Documentation Release 2.7 Michal Čihar July 10, 2016 Powered by TCPDF (www.tcpdf.org) 1 About Weblate 3 1.1 Project goals 3 1.2 Project name 3 1.3 Project website 3 1.4 Authors 3 . 155 10.3 wlc.config ..... 156 10.4 wlc.main ..... 156 11 Changes ..... 157 11.1 weblate 2.7 ..... 157 11.2 weblate 2.6 ..... 157 11.3 weblate 2.5 ..... 157 11.4 weblate 2.4 ..... 159 directly from its git repository using the OpenShift Client Tools: rhc -aweblate app create -t python-2.7 --from-code https://github.com/nijel/weblate.gi 2. After installation everything should be preconfigured

Python API - Installation - wlc - wlc.config - wlc.main - Changes - weblate 2.7 - weblate 2.6 - weblate 2.5 - weblate 2.4 - weblate 2.3 - weblate 2.2 - weblate string matching the original one. ## See also Python string formatting [https://docs.python.org/2.7/library/stdtypes.html#string-formatting], Python brace format [https://docs.python.org/3.3/library/string Upgrade from 2.3 to 2.4 - Upgrade from 2.4 to 2.5 - Upgrade from 2.5 to 2.6 - Upgrade from 2.6 to 2.7 - Upgrading to Django 1.7 - Migrating from Pootle - Authentication - User registration

# web sockets Documentation Release 2.7 Aymeric Augustin November 18, 2015 Powered by TCPDF (www.tcpdf.org) 1 Installation 3 2 User guide 5 3 Contributing 7 3.1 Getting started 7 3.2 Cheat sheet support connecting through a proxy. ### 3.6 Changelog ##### 3.6.1 3.0 In development ##### 3.6.2 2.7 • Added compatibility with Python 3.5. • Refreshed documentation. ##### 3.6.3 2.6 • Added local

### Scrapy 1.8 documentation Scrapy is a fast high-level web crawling [https://en.wikipedia.org/wiki/Web_crawler] and web scraping [https://en.wikipedia.org/wiki/Web_scraping] framework, used to crawl the scrapy tag [https://stackoverflow.com/tags/scrapy]. - Ask or search questions in the Scrapy subreddit [https://www.reddit.com/r/scrapy/]. • Search for questions on the archives of the scrapy-users com/forum/#!forum/scrapy-users]. • Ask a question in the #scrapy IRC channel, - Report bugs with Scrapy in our issue tracker [https://github.com/scrapy/scrapy/issues]. ## First steps ## Scrapy at a glance

Scrapy Documentation Release 0.20.2 Scrapy developers September 18, 2014 Contents 1 Getting help 3 2 First steps 5 2.1 Scrapy at a glance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.3 Scrapy Tutorial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 3.7 Scrapy shell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Scrapy Documentation Release 1.0.7 Scrapy developers Aug 14, 2017 First steps 1 Getting help 3 2 First steps 5 2.1 Scrapy at a glance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.3 Scrapy Tutorial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 3.6 Scrapy shell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .