Increment the value of the given stats key, by the given count, assuming the start value given (when it’s 54 Chapter 4. Built-in services Scrapy Documentation, Release 0.9 not set). If spider is not given the mail, 57 scrapy.management.telnet, 59 scrapy.selector, 34 scrapy.spider, 23 scrapy.stats.collector, 54 scrapy.stats.collector.simpledb, 56 scrapy.stats.signals, 56 scrapy.telnet, 100 scrapy.utils.trackref contrib.loader.ItemLoader method), 40 get_stats() (scrapy.stats.collector.StatsCollector method), 54 get_target() (scrapy.contrib.webservice.enginestatus.scrapy.webservice.JsonRpcResource method), 64

*processors, **kwargs) Similar to add_xpath() but replaces collected data instead of adding it. 54 Chapter 3. Basic concepts Scrapy Documentation, Release 0.24.6 get_css(css, *processors, **kwargs) [follow] INFO: Crawled 410 pages (at 4020 pages/min), scraped 0 items (at 0 items/min) 2013-05-16 13:08:54-0300 [follow] INFO: Crawled 474 pages (at 3840 pages/min), scraped 0 items (at 0 items/min) 2013-05-16 ItemLoader method), 55 add_value() (scrapy.contrib.loader.ItemLoader method), 54 add_xpath() (scrapy.contrib.loader.ItemLoader method), 54 adjust_request_args() (scrapy.contracts.Contract method), 95 AJAXCRAWL_ENABLED

Pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 3.9 Feed exports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . called when the spider is closed. Parameters spider (BaseSpider object) – the spider which was closed 54 Chapter 3. Scraping basics Scrapy Documentation, Release 0.12.0 3.8.2 Item pipeline example Price you can pass a custom version with the --version option: scrapy deploy scrapyd -p project1 --version 54 Also, if you use Mercurial for tracking your project source code, you can use HG for the version which

Loaders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 3.6 Scrapy shell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . field; the Item Loader will know how to “join” those values later using a proper processing function. 54 Chapter 3. Basic concepts Scrapy Documentation, Release 1.1.3 Here is a typical Item Loader usage [scrapy] INFO: Crawled 410 pages (at 4020 pages/min), ˓→scraped 0 items (at 0 items/min) 2013-05-16 13:08:54-0300 [scrapy] INFO: Crawled 474 pages (at 3840 pages/min), ˓→scraped 0 items (at 0 items/min) 2013-05-16

very useful for scraping. In particular, its Inspect Element [http://www.youtube.com/watch? v=-pT_pDe54aA] feature comes very handy when you need to construct the XPaths for extracting data because it allows Firebug comes with a very useful feature called Inspect Element [http://www.youtube.com/watch?v=-pT_pDe54aA] which allows you to inspect the HTML code of the different page elements just by hovering your mouse In the following screenshot you can see the Inspect Element [http://www.youtube.com/watch?v=-pT_pDe54aA] tool in action. At first sight, we can see that the directory is divided in categories, which are

very useful for scraping. In particular, its Inspect Element [http://www.youtube.com/watch? v=-pT_pDe54aA] feature comes very handy when you need to construct the XPaths for extracting data because it allows Firebug comes with a very useful feature called Inspect Element [http://www.youtube.com/watch?v=-pT_pDe54aA] which allows you to inspect the HTML code of the different page elements just by hovering your mouse In the following screenshot you can see the Inspect Element [http://www.youtube.com/watch?v=-pT_pDe54aA] tool in action. At first sight, we can see that the directory is divided in categories, which are

dicts. A separate class is used to support the item declaration syntax based on class attributes. 54 Chapter 3. Basic concepts Scrapy Documentation, Release 1.0.7 Item Loaders Item Loaders provide [scrapy] INFO: Crawled 410 pages (at 4020 pages/min), ˓→scraped 0 items (at 0 items/min) 2013-05-16 13:08:54-0300 [scrapy] INFO: Crawled 474 pages (at 3840 pages/min), ˓→scraped 0 items (at 0 items/min) 2013-05-16 setting, 74 FEED_URI setting, 73 fetch command, 26 Field (class in scrapy.item), 54 fields (scrapy.item.Item attribute), 54 fields_to_export (scrapy.exporters.BaseItemExporter at- tribute), 189 FILES_EXPIRES

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 3.5 Item Loaders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . tracks Item instances to help finding memory leaks (see Debugging memory leaks with trackref), etc. 54 Chapter 3. Basic concepts Scrapy Documentation, Release 1.3.3 Declaring Items Items are declared logstats] INFO: Crawled 254 pages (at 3360 ˓→pages/min), scraped 0 items (at 0 items/min) 2016-12-16 21:18:54 [scrapy.extensions.logstats] INFO: Crawled 302 pages (at 2880 ˓→pages/min), scraped 0 items (at 0

shell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 3.8 Item Pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . working on Unix systems (where IPython excels). See the IPython installation guide for more info. 54 Chapter 3. Basic concepts Scrapy Documentation, Release 0.18.4 3.7.1 Launch the shell To launch [follow] INFO: Crawled 410 pages (at 4020 pages/min), scraped 0 items (at 0 items/min) 2013-05-16 13:08:54-0300 [follow] INFO: Crawled 474 pages (at 3840 pages/min), scraped 0 items (at 0 items/min) 2013-05-16

very useful for scraping. In particular, its Inspect Element [http://www.youtube.com/watch? v=-pT_pDe54aA] feature comes very handy when you need to construct the XPaths for extracting data because it allows Firebug comes with a very useful feature called Inspect Element [http://www.youtube.com/watch?v=-pT_pDe54aA] which allows you to inspect the HTML code of the different page elements just by hovering your mouse In the following screenshot you can see the Inspect Element [http://www.youtube.com/watch?v=-pT_pDe54aA] tool in action. At first sight, we can see that the directory is divided in categories, which are