Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 6.4 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . documenta- tion for more info): •URLLENGTH_LIMIT - The maximum URL length to allow for crawled URLs. 6.4 Extensions The extensions framework provides a mechanism for inserting your own custom functionality ject): def __init__(self): dispatcher.connect(self.spider_opened, signal=signals.spider_opened) 6.4. Extensions 119 Scrapy Documentation, Release 0.12.0 dispatcher.connect(self.spider_closed, signal=signals

Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 6.4 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . documenta- tion for more info): •URLLENGTH_LIMIT - The maximum URL length to allow for crawled URLs. 6.4 Extensions The extensions framework provides a mechanism for inserting your own custom functionality at 0x9e272ac>, 'WebService': , ... 6.4. Extensions 123 Scrapy Documentation, Release 0.14.4 disabled A dict with the disabled extensions

Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 6.4 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Documentation, Release 0.16.5 •URLLENGTH_LIMIT - The maximum URL length to allow for crawled URLs. 6.4 Extensions The extensions framework provides a mechanism for inserting your own custom functionality (ie. those included in the EXTENSIONS_BASE setting) you must set its order to None. For example: 6.4. Extensions 133 Scrapy Documentation, Release 0.16.5 EXTENSIONS = { 'scrapy.contrib.corestats.CoreStats':

Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 6.4 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . documenta- tion for more info): •URLLENGTH_LIMIT - The maximum URL length to allow for crawled URLs. 6.4 Extensions The extensions framework provides a mechanism for inserting your own custom functionality all built-in extensions enabled by default) defines all the extensions with the same order (500). 6.4. Extensions 133 Scrapy Documentation, Release 0.22.0 6.4.3 Available, enabled and disabled extensions

Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 6.4 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . documenta- tion for more info): •URLLENGTH_LIMIT - The maximum URL length to allow for crawled URLs. 6.4 Extensions The extensions framework provides a mechanism for inserting your own custom functionality extensions loaded A boolean which is True if extensions are already loaded or False if they’re not. 6.4. Extensions 99 Scrapy Documentation, Release 0.9 enabled A dict with the enabled extensions. The

Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 6.4 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . documenta- tion for more info): •URLLENGTH_LIMIT - The maximum URL length to allow for crawled URLs. 6.4 Extensions The extensions framework provides a mechanism for inserting your own custom functionality Google Sitemaps would use settings like GOOGLE- SITEMAP_ENABLED, GOOGLESITEMAP_DEPTH, and so on. 6.4. Extensions 137 Scrapy Documentation, Release 0.24.6 6.4.2 Loading & activating extensions Extensions

Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 6.4 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . documenta- tion for more info): •URLLENGTH_LIMIT - The maximum URL length to allow for crawled URLs. 6.4 Extensions The extensions framework provides a mechanism for inserting your own custom functionality NotConfigured exception, the extension will be disabled. Oth- erwise, the extension will be enabled. 6.4. Extensions 129 Scrapy Documentation, Release 0.18.4 Sample extension Here we will implement a simple

Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 6.4 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . documenta- tion for more info): •URLLENGTH_LIMIT - The maximum URL length to allow for crawled URLs. 6.4 Extensions The extensions framework provides a mechanism for inserting your own custom functionality NotConfigured exception, the extension will be disabled. Oth- erwise, the extension will be enabled. 6.4. Extensions 131 Scrapy Documentation, Release 0.20.2 Sample extension Here we will implement a simple

Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200 6.4 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . to allow for crawled URLs. 206 Chapter 6. Extending Scrapy Scrapy Documentation, Release 1.7.4 6.4 Extensions The extensions framework provides a mechanism for inserting your own custom functionality its order to None. For example: EXTENSIONS = { 'scrapy.extensions.corestats.CoreStats': None, } 6.4. Extensions 207 Scrapy Documentation, Release 1.7.4 6.4.5 Writing your own extension Each extension

Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 6.4 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . documentation for more info): • URLLENGTH_LIMIT - The maximum URL length to allow for crawled URLs. 6.4 Extensions The extensions framework provides a mechanism for inserting your own custom functionality are just regular classes that are instantiated at Scrapy startup, when extensions are initialized. 6.4. Extensions 225 Scrapy Documentation, Release 1.8.4 6.4.1 Extension settings Extensions use the