extract_first(), } next_page = response.css('li.next a::attr("href")').extract_first() if next_page is not None: yield response.follow(next_page, self.parse) Put this in a text Selector, yield a Python dict with the extracted quote text and author, look for a link to the next page and schedule another request using the same parse method as callback. Here you notice one of the w3lib [https://pypi.python.org/pypi/w3lib], a multi-purpose helper for dealing with URLs and web page encodings twisted [https://twistedmatrix.com/], an asynchronous networking framework cryptography

get(), } (continues on next page) 5 Scrapy Documentation, Release 1.7.4 (continued from previous page) next_page = response.css('li.next a::attr("href")').get() if next_page is not None: yield response response.follow(next_page, self.parse) Put this in a text file, name it to something like quotes_spider.py and run the spider using the runspider command: scrapy runspider quotes_spider.py -o quotes.json Selector, yield a Python dict with the extracted quote text and author, look for a link to the next page and schedule another request using the same parse method as callback. Here you notice one of the

get(), } (continues on next page) 5 Scrapy Documentation, Release 1.6.0 (continued from previous page) next_page = response.css('li.next a::attr("href")').get() if next_page is not None: yield response response.follow(next_page, self.parse) Put this in a text file, name it to something like quotes_spider.py and run the spider using the runspider command: scrapy runspider quotes_spider.py -o quotes.json Selector, yield a Python dict with the extracted quote text and author, look for a link to the next page and schedule another request using the same parse method as callback. Here you notice one of the

quote.xpath('span/small/text()').get(), } next_page = response.css('li.next a::attr("href")').get() if next_page is not None: yield response.follow(next_page, self.parse) Put this in a text file, name it Selector, yield a Python dict with the extracted quote text and author, look for a link to the next page and schedule another request using the same parse method as callback. Here you notice one of the extraction library written on top of lxml, • w3lib, a multi-purpose helper for dealing with URLs and web page encodings • twisted, an asynchronous networking framework • cryptography and pyOpenSSL, to deal

quote.css('span.text::text').get(), } next_page = response.css('li.next a::attr("href")').get() if next_page is not None: yield response.follow(next_page, self.parse) Put this in a text file, name it Selector, yield a Python dict with the extracted quote text and author, look for a link to the next page and schedule another request using the same parse method as callback. Here you notice one of the extraction library written on top of lxml, • w3lib, a multi-purpose helper for dealing with URLs and web page encodings • twisted, an asynchronous networking framework • cryptography and pyOpenSSL, to deal

quote.css('span.text::text').get(), } next_page = response.css('li.next a::attr("href")').get() if next_page is not None: yield response.follow(next_page, self.parse) Put this in a text file, name it Selector, yield a Python dict with the extracted quote text and author, look for a link to the next page and schedule another request using the same parse method as callback. Here you notice one of the extraction library written on top of lxml, • w3lib, a multi-purpose helper for dealing with URLs and web page encodings • twisted, an asynchronous networking framework • cryptography and pyOpenSSL, to deal

(continues on next page) 5 Scrapy Documentation, Release 1.5.2 (continued from previous page) next_page = response.css('li.next a::attr("href")').extract_first() if next_page is not None: yield response response.follow(next_page, self.parse) Put this in a text file, name it to something like quotes_spider.py and run the spider using the runspider command: scrapy runspider quotes_spider.py -o quotes Selector, yield a Python dict with the extracted quote text and author, look for a link to the next page and schedule another request using the same parse method as callback. Here you notice one of the

quote.css('span.text::text').get(), } next_page = response.css('li.next a::attr("href")').get() if next_page is not None: yield response.follow(next_page, self.parse) Put this in a text file, name it Selector, yield a Python dict with the extracted quote text and author, look for a link to the next page and schedule another request using the same parse method as callback. Here you notice one of the extraction library written on top of lxml, • w3lib, a multi-purpose helper for dealing with URLs and web page encodings • twisted, an asynchronous networking framework • cryptography and pyOpenSSL, to deal

quote.css('span.text::text').get(), } next_page = response.css('li.next a::attr("href")').get() if next_page is not None: yield response.follow(next_page, self.parse) Put this in a text file, name it Selector, yield a Python dict with the extracted quote text and author, look for a link to the next page and schedule another request using the same parse method as callback. Here you notice one of the extraction library written on top of lxml, • w3lib, a multi-purpose helper for dealing with URLs and web page encodings • twisted, an asynchronous networking framework • cryptography and pyOpenSSL, to deal

quote.css('span.text::text').get(), } next_page = response.css('li.next a::attr("href")').get() if next_page is not None: yield response.follow(next_page, self.parse) Put this in a text file, name it Selector, yield a Python dict with the extracted quote text and author, look for a link to the next page and schedule another request using the same parse method as callback. Here you notice one of the extraction library written on top of lxml, • w3lib, a multi-purpose helper for dealing with URLs and web page encodings • twisted, an asynchronous networking framework • cryptography and pyOpenSSL, to deal