'mininova.org' allowed_domains = ['mininova.org'] start_urls = ['http://www.mininova.org/today'] rules = [Rule(SgmlLinkExtractor(allow=['/tor/\d+']), 'parse_torrent')] def parse_torrent(self, response): x = specify), this class supports a new attribute: rules Which is a list of one (or more) Rule objects. Each Rule defines a certain behaviour for crawling the site. Rules objects are described below . 3 3.2. Spiders 25 Scrapy Documentation, Release 0.9 Crawling rules class scrapy.contrib.spiders.Rule(link_extractor, callback=None, cb_kwargs=None, fol- low=None, process_links=None) link_extractor is

'mininova.org' allowed_domains = ['mininova.org'] start_urls = ['http://www.mininova.org/today'] rules = [Rule(SgmlLinkExtractor(allow=['/tor/\d+']), 'parse_torrent')] def parse_torrent(self, response): x = specify), this class supports a new attribute: rules Which is a list of one (or more) Rule objects. Each Rule defines a certain behaviour for crawling the site. Rules objects are described below. If according to the order they’re defined in this attribute. Crawling rules class scrapy.contrib.spiders.Rule(link_extractor, callback=None, cb_kwargs=None, fol- low=None, process_links=None, process_request=None)

allowed_domains = ['mininova.org'] start_urls = ['http://www.mininova.org/today'] rules = [Rule(SgmlLinkExtractor(allow=['/tor/\d+']), 'parse_torrent')] def parse_torrent(self, response): list of one (or more) Rule objects. Each Rule defines a certain behaviour for crawling the site. Rules objects are described below . Crawling rules class scrapy.contrib.spiders.Rule(link_extractor, callback=None follow is a boolean which specified if links should be followed from each response extracted with this rule. If callback is None follow defaults to True, otherwise it default to False. process_links is a callable

allowed_domains = ['mininova.org'] start_urls = ['http://www.mininova.org/today'] rules = [Rule(SgmlLinkExtractor(allow=['/tor/\d+']), 'parse_torrent')] def parse_torrent(self, response): specify), this class supports a new attribute: rules Which is a list of one (or more) Rule objects. Each Rule defines a certain behaviour for crawling the site. Rules objects are described below. If according to the order they’re defined in this attribute. Crawling rules class scrapy.contrib.spiders.Rule(link_extractor, callback=None, cb_kwargs=None, follow=None, process_links=None, process_request=None)

specify), this class supports a new attribute: rules Which is a list of one (or more) Rule objects. Each Rule defines a certain behaviour for crawling the site. Rules objects are described below. If object, a Request object, or an iterable containing any of them. Crawling rules class scrapy.spiders.Rule(link_extractor=None, callback=None, cb_kwargs=None, follow=None, process_links=None, process_request=None follow is a boolean which specifies if links should be followed from each response extracted with this rule. If callback is None follow defaults to True, otherwise it defaults to False. 3.2. Spiders 37 Scrapy

allowed_domains = ['mininova.org'] start_urls = ['http://www.mininova.org/today'] rules = [Rule(SgmlLinkExtractor(allow=['/tor/\d+']), 'parse_torrent')] def parse_torrent(self, response): specify), this class supports a new attribute: rules Which is a list of one (or more) Rule objects. Each Rule defines a certain behaviour for crawling the site. Rules objects are described below. If according to the order they’re defined in this attribute. Crawling rules class scrapy.contrib.spiders.Rule(link_extractor, callback=None, cb_kwargs=None, follow=None, process_links=None, process_request=None)

'mininova.org' allowed_domains = ['mininova.org'] start_urls = ['http://www.mininova.org/today'] rules = [Rule(SgmlLinkExtractor(allow=['/tor/\d+']), 'parse_torrent')] def parse_torrent(self, response): x = specify), this class supports a new attribute: rules Which is a list of one (or more) Rule objects. Each Rule defines a certain behaviour for crawling the site. Rules objects are described below. If according to the order they’re defined in this attribute. Crawling rules class scrapy.contrib.spiders.Rule(link_extractor, callback=None, cb_kwargs=None, fol- low=None, process_links=None, process_request=None)

specify), this class supports a new attribute: rules Which is a list of one (or more) Rule objects. Each Rule defines a certain behaviour for crawling the site. Rules objects are described below. If object, a Request object, or an iterable containing any of them. Crawling rules class scrapy.spiders.Rule(link_extractor=None, callback=None, cb_kwargs=None, follow=None, process_links=None, process_request=None follow is a boolean which specifies if links should be followed from each response extracted with this rule. If callback is None follow defaults to True, otherwise it defaults to False. process_links is a

specify), this class supports a new attribute: rules Which is a list of one (or more) Rule objects. Each Rule defines a certain behaviour for crawling the site. Rules objects are described below. If them. 3.2. Spiders 37 Scrapy Documentation, Release 2.1.0 Crawling rules class scrapy.spiders.Rule(link_extractor=None, callback=None, cb_kwargs=None, follow=None, process_links=None, process_request=None follow is a boolean which specifies if links should be followed from each response extracted with this rule. If callback is None follow defaults to True, otherwise it defaults to False. process_links is a

specify), this class supports a new attribute: rules Which is a list of one (or more) Rule objects. Each Rule defines a certain behaviour for crawling the site. Rules objects are described below. If them. 3.2. Spiders 37 Scrapy Documentation, Release 2.2.1 Crawling rules class scrapy.spiders.Rule(link_extractor=None, callback=None, cb_kwargs=None, follow=None, process_links=None, process_request=None follow is a boolean which specifies if links should be followed from each response extracted with this rule. If callback is None follow defaults to True, otherwise it defaults to False. process_links is a