Crawling rules class scrapy.spiders.Rule(link_extractor, callback=None, cb_kwargs=None, follow=None, pro- cess_links=None, process_request=None) link_extractor is a Link Extractor object which defines how processor which is constructed from the composition of the given functions, similar to the Compose pro- cessor. The difference with this processor is the way internal results are passed among functions (or ../ when relevant). scrapy shell index.html will not work as one might expect (and this is by design, not a bug). Because shell favors HTTP URLs over File URIs, and index.html being syntactically similar

Crawling rules class scrapy.spiders.Rule(link_extractor, callback=None, cb_kwargs=None, follow=None, pro- cess_links=None, process_request=None) link_extractor is a Link Extractor object which defines how processor which is constructed from the composition of the given functions, similar to the Compose pro- cessor. The difference with this processor is the way internal results are passed among functions (or ../ when relevant). scrapy shell index.html will not work as one might expect (and this is by design, not a bug). Because shell favors HTTP URLs over File URIs, and index.html being syntactically similar

class scrapy.contrib.linkextractors.sgml.BaseSgmlLinkExtractor(tag=”a”, attr=”href”, unique=False, pro- cess_value=None) The purpose of this Link Extractor is only to serve as a base class for the SgmlLinkExtractor processor which is constructed from the composition of the given functions, similar to the Compose pro- cessor. The difference with this processor is the way internal results are passed among functions class scrapy.contrib.linkextractors.sgml.BaseSgmlLinkExtractor(tag=”a”, attr=”href”, unique=False, pro- cess_value=None) The purpose of this Link Extractor is only to serve as a base class for the SgmlLinkExtractor

Crawling rules class scrapy.spiders.Rule(link_extractor, callback=None, cb_kwargs=None, follow=None, pro- cess_links=None, process_request=None) link_extractor is a Link Extractor object which defines how processor which is constructed from the composition of the given functions, similar to the Compose pro- cessor. The difference with this processor is the way internal results are passed among functions (or ../ when relevant). scrapy shell index.html will not work as one might expect (and this is by design, not a bug). Because shell favors HTTP URLs over File URIs, and index.html being syntactically similar

processor which is constructed from the composition of the given functions, similar to the Compose pro- cessor. The difference with this processor is the way internal results are passed among functions deny_extensions=None, restrict_xpaths=(), tags=(‘a’, ‘area’), attrs=(‘href’, ), canonicalize=True, unique=True, pro- cess_value=None) LxmlLinkExtractor is the recommended link extractor with handy filtering options class scrapy.contrib.linkextractors.sgml.BaseSgmlLinkExtractor(tag=”a”, attr=”href”, unique=False, pro- cess_value=None) The purpose of this Link Extractor is only to serve as a base class for the SgmlLinkExtractor

Crawling rules class scrapy.spiders.Rule(link_extractor, callback=None, cb_kwargs=None, follow=None, pro- cess_links=None, process_request=None) link_extractor is a Link Extractor object which defines how processor which is constructed from the composition of the given functions, similar to the Compose pro- cessor. The difference with this processor is the way internal results are passed among functions (or ../ when relevant). scrapy shell index.html will not work as one might expect (and this is by design, not a bug). Because shell favors HTTP URLs over File URIs, and index.html being syntactically similar

class scrapy.contrib.linkextractors.sgml.BaseSgmlLinkExtractor(tag=”a”, attr=”href”, unique=False, pro- cess_value=None) The purpose of this Link Extractor is only to serve as a base class for the SgmlLinkExtractor processor which is constructed from the composition of the given functions, similar to the Compose pro- cessor. The difference with this processor is the way internal results are passed among functions web service contains several resources, defined in the WEBSERVICE_RESOURCES setting. Each resource pro- vides a different functionality. See Available JSON-RPC resources for a list of resources available

class scrapy.contrib.linkextractors.sgml.BaseSgmlLinkExtractor(tag=”a”, attr=”href”, unique=False, pro- cess_value=None) The purpose of this Link Extractor is only to serve as a base class for the SgmlLinkExtractor processor which is constructed from the composition of the given functions, similar to the Compose pro- cessor. The difference with this processor is the way internal results are passed among functions web service contains several resources, defined in the WEBSERVICE_RESOURCES setting. Each resource pro- vides a different functionality. See Available JSON-RPC resources for a list of resources available

processor which is constructed from the composition of the given functions, similar to the Compose pro- cessor. The difference with this processor is the way internal results are passed among functions class scrapy.contrib.linkextractors.sgml.BaseSgmlLinkExtractor(tag=”a”, attr=”href”, unique=False, pro- cess_value=None) The purpose of this Link Extractor is only to serve as a base class for the SgmlLinkExtractor web service contains several resources, defined in the WEBSERVICE_RESOURCES setting. Each resource pro- vides a different functionality. See Available JSON-RPC resources for a list of resources available

Crawling rules class scrapy.spiders.Rule(link_extractor, callback=None, cb_kwargs=None, follow=None, pro- cess_links=None, process_request=None) link_extractor is a Link Extractor object which defines how processor which is constructed from the composition of the given functions, similar to the Compose pro- cessor. The difference with this processor is the way internal results are passed among functions (or ../ when relevant). scrapy shell index.html will not work as one might expect (and this is by design, not a bug). Because shell favors HTTP URLs over File URIs, and index.html being syntactically similar