# Scrapy Documentation Release 0.16.5 Scrapinghub May 12, 2016 Powered by TCPDF (www.tcpdf.org) 1 Getting help 3 2 First steps 5 2.1 Scrapy at a glance 5 2.2 Installation guide 8 2.3 Scrapy Items 24 3.3 Spiders 28 3.4 Link Extractors 36 3.5 Selectors 38 3.6 Item Loaders 44 3.7 Scrapy shell 51 3.8 Item Pipeline 54 3.9 Feed exports 57 4 Built-in services 63 4.1 Logging 63 packages 102 5.11 Scrapy Service (scrapyd) 102 5.12 AutoThrottle extension ..... 113 5.13 Jobs: pausing and resuming crawls ..... 114 5.14 DjangoItem ..... 116 6 Extending Scrapy ..... 119 6.1

### Scrapy 2.6 documentation Scrapy is a fast high-level web crawling [https://en.wikipedia.org/wiki/Web_crawler] and web scraping [https://en.wikipedia.org/wiki/Web_scraping] framework, used to crawl the scrapy tag [https://stackoverflow.com/tags/scrapy]. • Ask or search questions in the Scrapy subreddit [https://www.reddit.com/r/scrapy/]. • Search for questions on the archives of the scrapy-users forum/scrapy-users]. • Ask a question in the #scrapy IRC channel. • Report bugs with Scrapy in our issue tracker [https://github.com/scrapy/scrapy/issues]. • Join the Discord community Scrapy Discord

### Scrapy 2.7 documentation Scrapy is a fast high-level web crawling [https://en.wikipedia.org/wiki/Web_crawler] and web scraping [https://en.wikipedia.org/wiki/Web_scraping] framework, used to crawl the scrapy tag [https://stackoverflow.com/tags/scrapy]. • Ask or search questions in the Scrapy subreddit [https://www.reddit.com/r/scrapy/]. • Search for questions on the archives of the scrapy-users forum/scrapy-users]. • Ask a question in the #scrapy IRC channel. • Report bugs with Scrapy in our issue tracker [https://github.com/scrapy/scrapy/issues]. • Join the Discord community Scrapy Discord

# Scrapy Documentation Release 2.11.1 Scrapy developers Feb 14, 2024 Powered by TCPDF (www.tcpdf.org) ## FIRST STEPS 1 Getting help 3 2 First steps 5 2.1 Scrapy at a glance 5 2.2 Installation guide 7 2.3 Scrapy Tutorial 11 2.4 Examples 23 3 Basic concepts 25 3.1 Command line tool 25 3.2 Spiders 35 3.3 Selectors 49 3.4 Items 68 3.5 Item Loaders 74 3.6 Scrapy shell 84 3 and resuming crawls ..... 224 5.14 Coroutines ..... 225 5.15 asyncio ..... 230 6 Extending Scrapy ..... 235 6.1 Architecture overview ..... 235 6.2 Add-ons ..... 238 6.3 Downloader Middleware

# Scrapy Documentation Release 0.9 Insophia May 12, 2016 Powered by TCPDF (www.tcpdf.org) 1 Getting help 3 2 First steps 5 2.1 Scrapy at a glance 5 2.2 Installation guide 7 2.3 Scrapy Tutorial 19 3.2 Spiders 23 3.3 Link Extractors 29 3.4 XPath Selectors 31 3.5 Item Loaders 36 3.6 Scrapy shell 44 3.7 Item Pipeline 47 4 Built-in services 51 4.1 Logging 51 4.2 Stats Collection Images 81 6 Extending Scrapy 87 6.1 Architecture overview 87 6.2 Downloader Middleware 89 6.3 Spider Middleware 94 6.4 Extensions 98 7 Reference 103 7.1 scrapy-ctl.py 103 7.2 Requests

### Scrapy 0.14 documentation This documentation contains everything you need to know about Scrapy. ## Getting help Having trouble? We’d like to help! • Try the FAQ – it’s got answers to some common archives of the scrapy-users mailing list [http://groups.google.com/group/scrapy-users/], or post a question [http://groups.google.com/group/scrapy-users/]. • Ask a question in the #scrapy IRC channel. • Report bugs with Scrapy in our issue tracker [https://github.com/scrapy/scrapy/issues]. ## First steps ## Scrapy at a glance Understand what Scrapy is and how it can help you. ## I nstallation guide

# Scrapy Documentation Release 2.0.1 Scrapy developers May 19, 2020 Powered by TCPDF (www.tcpdf.org) ## FIRST STEPS 1 Getting help 3 2 First steps 5 2.1 Scrapy at a glance 5 2.2 Installation guide 7 2.3 Scrapy Tutorial 11 2.4 Examples 22 3 Basic concepts 23 3.1 Command line tool 23 3.2 Spiders 33 3.3 Selectors 44 3.4 Items 61 3.5 Item Loaders 65 3.6 Scrapy shell 76 3 and resuming crawls ..... 193 5.14 Coroutines ..... 194 5.15 asyncio ..... 196 6 Extending Scrapy ..... 197 6.1 Architecture overview ..... 197 6.2 Downloader Middleware ..... 200 6.3 Spider

# Scrapy Documentation Release 2.7.1 Scrapy developers Nov 02, 2022 Powered by TCPDF (www.tcpdf.org) ## FIRST STEPS 1 Getting help 3 2 First steps 5 2.1 Scrapy at a glance 5 2.2 Installation guide 7 2.3 Scrapy Tutorial 11 2.4 Examples 23 3 Basic concepts 25 3.1 Command line tool 25 3.2 Spiders 35 3.3 Selectors 48 3.4 Items 66 3.5 Item Loaders 72 3.6 Scrapy shell 81 3 13 Jobs: pausing and resuming crawls 217 5.14 Coroutines 218 5.15 asyncio 222 6 Extending Scrapy 227 6.1 Architecture overview 227 6.2 Downloader Middleware 230 6.3 Spider Middleware 247

# Scrapy 0.20 documentation This documentation contains everything you need to know about Scrapy. ## Getting help Having trouble? We’d like to help! • Try the FAQ – it’s got answers to some common archives of the scrapy-users mailing list [http://groups.google.com/group/scrapy-users/], or post a question [http://groups.google.com/group/scrapy-users/]. • Ask a question in the #scrapy IRC channel. • Report bugs with Scrapy in our issue tracker [https://github.com/scrapy/scrapy/issues]. ## First steps ## Scrapy at a glance Understand what Scrapy is and how it can help you. ## I nstallation guide

Scrapy 1.2 documentation This documentation contains everything you need to know about Scrapy. Getting help Having trouble? We’d like to help! Try the FAQ – it’s got answers to some common questions. StackOverflow using the scrapy tag [https://stackoverflow.com/tags/scrapy], Search for information in the archives of the scrapy-users mailing list [https://groups.google.com/forum/#!forum/scrapy-users], or post com/forum/#!forum/scrapy-users]. Ask a question in the #scrapy IRC channel, Report bugs with Scrapy in our issue tracker [https://github.com/scrapy/scrapy/issues]. First steps Scrapy at a glance Understand