Firebug for scraping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 5.4 Debugging memory leaks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . elements. or tags which Therefer in page HTML sources may on Firebug inspects the live DOM 5.4 Debugging memory leaks In Scrapy, objects such as Requests, Responses and Items have a finite lifetime: leaks. It basically tracks the references to all live Requests, Responses, Item and Selector objects. 5.4. Debugging memory leaks 77 Scrapy Documentation, Release 0.9 To activate the trackref module, enable

Firebug for scraping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 5.4 Debugging memory leaks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . elements. or tags which Therefer in page HTML sources may on Firebug inspects the live DOM 5.4 Debugging memory leaks In Scrapy, objects such as Requests, Responses and Items have a finite lifetime: memory debugging (see below for more info). Both mechanisms must be used from the Telnet Console. 5.4. Debugging memory leaks 87 Scrapy Documentation, Release 0.12.0 5.4.1 Common causes of memory leaks

Firebug for scraping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 5.4 Debugging memory leaks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . elements. or tags which Therefer in page HTML sources may on Firebug inspects the live DOM 5.4 Debugging memory leaks In Scrapy, objects such as Requests, Responses and Items have a finite lifetime: scrapy.http.Response • scrapy.item.Item • scrapy.selector.XPathSelector • scrapy.spider.BaseSpider 5.4. Debugging memory leaks 89 Scrapy Documentation, Release 0.14.4 • scrapy.selector.document.Libxml2Document

Contracts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 5.4 Common Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . self.args: if header not in response.headers: raise ContractFail(’X-CustomHeader not present’) 5.4 Common Practices This section documents common practices when using Scrapy. These are things that top of the Twisted asynchronous networking library, so you need run it inside the Twisted reactor. 5.4. Common Practices 91 Scrapy Documentation, Release 0.22.0 Note that you will also have to shutdown

Contracts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 5.4 Common Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . self.args: if header not in response.headers: raise ContractFail(’X-CustomHeader not present’) 5.4 Common Practices This section documents common practices when using Scrapy. These are things that top of the Twisted asynchronous networking library, so you need run it inside the Twisted reactor. 5.4. Common Practices 89 Scrapy Documentation, Release 0.20.2 Note that you will also have to shutdown

Contracts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 5.4 Common Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . self.args: if header not in response.headers: raise ContractFail('X-CustomHeader not present') 5.4 Common Practices This section documents common practices when using Scrapy. These are things that the API to run Scrapy from a script, instead of the typical way of running Scrapy via scrapy crawl. 5.4. Common Practices 95 Scrapy Documentation, Release 0.24.6 Remember that Scrapy is built on top of

Contracts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 5.4 Common Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . is running 5.4 Common Practices This section documents common practices when using Scrapy. These are things that cover many topics and don’t often fall into any other specific section. 5.4. Common Practices configure_logging class MySpider1(scrapy.Spider): # Your first spider definition ... (continues on next page) 5.4. Common Practices 163 Scrapy Documentation, Release 1.8.4 (continued from previous page) class MySpider2(scrapy

Contracts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 5.4 Common Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . adjustments when a check is running 5.3. Spiders Contracts 165 Scrapy Documentation, Release 2.2.1 5.4 Common Practices This section documents common practices when using Scrapy. These are things that twisted.internet import reactor from scrapy.crawler import CrawlerRunner (continues on next page) 5.4. Common Practices 167 Scrapy Documentation, Release 2.2.1 (continued from previous page) from scrapy

Contracts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 5.4 Common Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . __init__(self): if os.environ.get('SCRAPY_CHECK'): pass # Do some scraper adjustments when a check is running 5.4 Common Practices This section documents common practices when using Scrapy. These are things that process.crawl(MySpider) process.start() # the script will block here until the crawling is finished 5.4. Common Practices 167 Scrapy Documentation, Release 2.4.1 Define settings within dictionary in CrawlerProcess

Contracts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 5.4 Common Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . __init__(self): if os.environ.get('SCRAPY_CHECK'): pass # Do some scraper adjustments when a check is running 5.4 Common Practices This section documents common practices when using Scrapy. These are things that process.crawl(MySpider) process.start() # the script will block here until the crawling is finished 5.4. Common Practices 167 Scrapy Documentation, Release 2.3.0 Define settings within dictionary in CrawlerProcess