# Scrapy Documentation Release 2.7.1 Scrapy developers Nov 02, 2022 Powered by TCPDF (www.tcpdf.org) ## FIRST STEPS 1 Getting help 3 2 First steps 5 2.1 Scrapy at a glance 5 2.2 Installation guide 7 2.3 Scrapy Tutorial 11 2.4 Examples 23 3 Basic concepts 25 3.1 Command line tool 25 3.2 Spiders 35 3.3 Selectors 48 3.4 Items 66 3.5 Item Loaders 72 3.6 Scrapy shell 81 3 13 Jobs: pausing and resuming crawls 217 5.14 Coroutines 218 5.15 asyncio 222 6 Extending Scrapy 227 6.1 Architecture overview 227 6.2 Downloader Middleware 230 6.3 Spider Middleware 247

### Scrapy 1.8 documentation Scrapy is a fast high-level web crawling [https://en.wikipedia.org/wiki/Web_crawler] and web scraping [https://en.wikipedia.org/wiki/Web_scraping] framework, used to crawl the scrapy tag [https://stackoverflow.com/tags/scrapy]. - Ask or search questions in the Scrapy subreddit [https://www.reddit.com/r/scrapy/]. • Search for questions on the archives of the scrapy-users com/forum/#!forum/scrapy-users]. • Ask a question in the #scrapy IRC channel, - Report bugs with Scrapy in our issue tracker [https://github.com/scrapy/scrapy/issues]. ## First steps ## Scrapy at a glance

Scrapy Documentation Release 0.20.2 Scrapy developers September 18, 2014 Contents 1 Getting help 3 2 First steps 5 2.1 Scrapy at a glance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.3 Scrapy Tutorial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 3.7 Scrapy shell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Scrapy Documentation Release 1.0.7 Scrapy developers Aug 14, 2017 First steps 1 Getting help 3 2 First steps 5 2.1 Scrapy at a glance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.3 Scrapy Tutorial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 3.6 Scrapy shell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

# Scrapy Documentation Release 1.7.4 Scrapy developers Nov 01, 2019 Powered by TCPDF (www.tcpdf.org) 1 Getting help 3 2 First steps 5 2.1 Scrapy at a glance 5 2.2 Installation guide 7 2.3 3 Scrapy Tutorial 11 2.4 Examples 22 3 Basic concepts 23 3.1 Command line tool 23 3.2 Spiders 32 3.3 Selectors 44 3.4 Items 61 3.5 Item Loaders 64 3.6 Scrapy shell 75 3.7 Item Pipeline 176 5.12 Benchmarking ..... 179 5.13 Jobs: pausing and resuming crawls ..... 180 6 Extending Scrapy ..... 183 6.1 Architecture overview ..... 183 6.2 Downloader Middleware ..... 186 6.3 Spider

### Scrapy 2.2 documentation Scrapy is a fast high-level web crawling [https://en.wikipedia.org/wiki/Web_crawler] and web scraping [https://en.wikipedia.org/wiki/Web_scraping] framework, used to crawl the scrapy tag [https://stackoverflow.com/tags/scrapy]. • Ask or search questions in the Scrapy subreddit [https://www.reddit.com/r/scrapy/]. • Search for questions on the archives of the scrapy-users com/forum/#!forum/scrapy-users]. • Ask a question in the #scrapy IRC channel. • Report bugs with Scrapy in our issue tracker [https://github.com/scrapy/scrapy/issues]. ## First steps ## Scrapy at a glance

### Scrapy 2.4 documentation Scrapy is a fast high-level web crawling [https://en.wikipedia.org/wiki/Web_crawler] and web scraping [https://en.wikipedia.org/wiki/Web_scraping] framework, used to crawl the scrapy tag [https://stackoverflow.com/tags/scrapy]. • Ask or search questions in the Scrapy subreddit [https://www.reddit.com/r/scrapy/]. • Search for questions on the archives of the scrapy-users com/forum/#!forum/scrapy-users]. • Ask a question in the #scrapy IRC channel. • Report bugs with Scrapy in our issue tracker [https://github.com/scrapy/scrapy/issues]. ## First steps ## Scrapy at a glance

# Scrapy Documentation Release 2.3.0 Scrapy developers Aug 04, 2020 Powered by TCPDF (www.tcpdf.org) ## FIRST STEPS 1 Getting help 3 2 First steps 5 2.1 Scrapy at a glance 5 2.2 Installation guide 7 2.3 Scrapy Tutorial 11 2.4 Examples 23 3 Basic concepts 25 3.1 Command line tool 25 3.2 Spiders 35 3.3 Selectors 46 3.4 Items 64 3.5 Item Loaders 70 3.6 Scrapy shell 78 3 and resuming crawls ..... 200 5.14 Coroutines ..... 202 5.15 asyncio ..... 203 6 Extending Scrapy ..... 205 6.1 Architecture overview ..... 205 6.2 Downloader Middleware ..... 208 6.3 Spider

# Scrapy Documentation Release 1.8.4 Scrapy developers Feb 14, 2024 Powered by TCPDF (www.tcpdf.org) ## FIRST STEPS 1 Getting help 3 2 First steps 5 2.1 Scrapy at a glance 5 2.2 Installation guide 7 2.3 Scrapy Tutorial 11 2.4 Examples 23 3 Basic concepts 25 3.1 Command line tool 25 3.2 Spiders 35 3.3 Selectors 46 3.4 Items 65 3.5 Item Loaders 69 3.6 Scrapy shell 80 3 191 5.12 Benchmarking ..... 193 5.13 Jobs: pausing and resuming crawls ..... 195 6 Extending Scrapy ..... 199 6.1 Architecture overview ..... 199 6.2 Downloader Middleware ..... 202 6.3 Spider

# Scrapy 2.11 documentation Scrapy is a fast high-level web crawling [https://en.wikipedia.org/wiki/Web_crawler] and web scraping [https://en.wikipedia.org/wiki/Web_scraping] framework, used to crawl the scrapy tag [https://stackoverflow.com/tags/scrapy]. • Ask or search questions in the Scrapy subreddit [https://www.reddit.com/r/scrapy/]. • Search for questions on the archives of the scrapy-users forum/scrapy-users]. • Ask a question in the #scrapy IRC channel, • Report bugs with Scrapy in our issue tracker [https://github.com/scrapy/scrapy/issues]. • Join the Discord community Scrapy Discord