Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172 6.3 Spider Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . example spider In order to show you what Scrapy brings to the table, we’ll walk you through an example of a Scrapy Spider using the simplest way to run a spider. Here’s the code for a spider that scrapes http://quotes.toscrape.com, following the pagi- nation: import scrapy class QuotesSpider(scrapy.Spider): name = "quotes" start_urls = [ 'http://quotes.toscrape.com/tag/humor/', ] def parse(self, response):

the Scrapy architecture. Downloader Middleware Customize how pages get requested and downloaded. Spider Middleware Customize the input and output of your spiders. Extensions Add any custom functionality name = Field() description = Field() size = Field() Write a Spider to extract the data The next thing is to write a Spider which defines the start URL (http://www.mininova.org/today), the rules information about XPath see the XPath reference [http://www.w3.org/TR/xpath]. Finally, here’s the spider code: class MininovaSpider(CrawlSpider): name = 'mininova.org' allowed_domains = ['mininova

Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 i 6.3 Spider Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . size = Field() 5 Scrapy Documentation, Release 0.14.4 2.1.3 Write a Spider to extract the data The next thing is to write a Spider which defines the start URL (http://www.mininova.org/today), the rules XPath reference. 6 Chapter 2. First steps Scrapy Documentation, Release 0.14.4 Finally, here’s the spider code: class MininovaSpider(CrawlSpider): name = 'mininova.org' allowed_domains = ['mininova.org']

Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241 6.4 Spider Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . example spider In order to show you what Scrapy brings to the table, we’ll walk you through an example of a Scrapy Spider using the simplest way to run a spider. Here’s the code for a spider that scrapes https://quotes.toscrape.com, following the pagi- nation: import scrapy class QuotesSpider(scrapy.Spider): name = "quotes" start_urls = [ "https://quotes.toscrape.com/tag/humor/", ] def parse(self, response):

Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 6.3 Spider Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . example spider In order to show you what Scrapy brings to the table, we’ll walk you through an example of a Scrapy Spider using the simplest way to run a spider. Here’s the code for a spider that scrapes http://quotes.toscrape.com, following the pagi- nation: import scrapy class QuotesSpider(scrapy.Spider): name = "quotes" start_urls = [ 'http://quotes.toscrape.com/tag/humor/', ] def parse(self, response):

Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222 6.3 Spider Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . example spider In order to show you what Scrapy brings to the table, we’ll walk you through an example of a Scrapy Spider using the simplest way to run a spider. Here’s the code for a spider that scrapes https://quotes.toscrape.com, following the pagi- nation: import scrapy class QuotesSpider(scrapy.Spider): name = 'quotes' start_urls = [ 'https://quotes.toscrape.com/tag/humor/', ] def parse(self, response):

Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180 6.3 Spider Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . example spider In order to show you what Scrapy brings to the table, we’ll walk you through an example of a Scrapy Spider using the simplest way to run a spider. Here’s the code for a spider that scrapes http://quotes.toscrape.com, following the pagi- nation: import scrapy class QuotesSpider(scrapy.Spider): name = 'quotes' start_urls = [ 'http://quotes.toscrape.com/tag/humor/', ] def parse(self, response):

Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234 6.3 Spider Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . example spider In order to show you what Scrapy brings to the table, we’ll walk you through an example of a Scrapy Spider using the simplest way to run a spider. Here’s the code for a spider that scrapes https://quotes.toscrape.com, following the pagi- nation: import scrapy class QuotesSpider(scrapy.Spider): name = "quotes" start_urls = [ "https://quotes.toscrape.com/tag/humor/", ] def parse(self, response):

Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232 6.3 Spider Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . example spider In order to show you what Scrapy brings to the table, we’ll walk you through an example of a Scrapy Spider using the simplest way to run a spider. Here’s the code for a spider that scrapes https://quotes.toscrape.com, following the pagi- nation: import scrapy class QuotesSpider(scrapy.Spider): name = 'quotes' start_urls = [ 'https://quotes.toscrape.com/tag/humor/', ] def parse(self, response):

Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 6.3 Spider Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . example spider In order to show you what Scrapy brings to the table, we’ll walk you through an example of a Scrapy Spider using the simplest way to run a spider. Here’s the code for a spider that scrapes http://quotes.toscrape.com, following the pagi- nation: import scrapy class QuotesSpider(scrapy.Spider): name = "quotes" start_urls = [ 'http://quotes.toscrape.com/tag/humor/', ] def parse(self, response):