our spider. You can see a log line for each URL defined in start_urls. Because these URLs are the starting ones, they have no referrers, which is shown at the end of the log line, where it says (referer: org/Computers/Programming/Languages/Python/Books/ This is what the shell looks like: [ ... Scrapy log here ... ] [s] Available Scrapy objects: [s] 2010-08-19 21:45:59-0300 [default] INFO: Spider closed each depth level Usage example: $ scrapy parse http://www.example.com/ -c parse_item [ ... scrapy log lines crawling example.com spider ... ] >>> STATUS DEPTH LEVEL 1 <<< # Scraped Items ------------

our spider. You can see a log line for each URL defined in start_urls. Because these URLs are the starting ones, they have no referrers, which is shown at the end of the log line, where it says (referer: org/Computers/Programming/Languages/Python/Books/ This is what the shell looks like: [ ... Scrapy log here ... ] [s] Available Scrapy objects: [s] 2010-08-19 21:45:59-0300 [default] INFO: Spider closed each depth level Usage example: $ scrapy parse http://www.example.com/ -c parse_item [ ... scrapy log lines crawling example.com spider ... ] >>> STATUS DEPTH LEVEL 1 <<< # Scraped Items ------------

our spider. You can see a log line for each URL defined in start_urls. Because these URLs are the starting ones, they have no referrers, which is shown at the end of the log line, where it says (referer: org/Computers/Programming/Languages/Python/Books/ This is what the shell looks like: [ ... Scrapy log here ... ] [s] Available Scrapy objects: [s] 2010-08-19 21:45:59-0300 [default] INFO: Spider closed each depth level Usage example: $ scrapy parse http://www.example.com/ -c parse_item [ ... scrapy log lines crawling example.com spider ... ] >>> STATUS DEPTH LEVEL 1 <<< # Scraped Items -----------

split("/")[-2] filename = 'quotes-%s.html' % page with open(filename, 'wb') as f: f.write(response.body) self.log('Saved file %s' % filename) As you can see, our Spider subclasses scrapy.Spider and defines some attributes instead: scrapy shell "http://quotes.toscrape.com/page/1/" You will see something like: [ ... Scrapy log here ... ] 2016-09-19 12:09:27 [scrapy.core.engine] DEBUG: Crawled (200) log: 2016-09-19 18:57:19 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes. ˓→toscrape.com/page/1/>

url.split("/")[-2] filename = f"quotes-{page}.html" Path(filename).write_bytes(response.body) self.log(f"Saved file {filename}") As you can see, our Spider subclasses scrapy.Spider and defines some attributes instead: scrapy shell "https://quotes.toscrape.com/page/1/" You will see something like: [ ... Scrapy log here ... ] 2016-09-19 12:09:27 [scrapy.core.engine] DEBUG: Crawled (200) log: 2016-09-19 18:57:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://quotes. ˓→toscrape.com/page/1/>

url.split("/")[-2] filename = f"quotes-{page}.html" Path(filename).write_bytes(response.body) self.log(f"Saved file {filename}") As you can see, our Spider subclasses scrapy.Spider and defines some attributes instead: scrapy shell "https://quotes.toscrape.com/page/1/" You will see something like: [ ... Scrapy log here ... ] 2016-09-19 12:09:27 [scrapy.core.engine] DEBUG: Crawled (200) log: 2016-09-19 18:57:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://quotes. ˓→toscrape.com/page/1/>

url.split("/")[-2] filename = f"quotes-{page}.html" Path(filename).write_bytes(response.body) self.log(f"Saved file {filename}") As you can see, our Spider subclasses scrapy.Spider and defines some attributes instead: scrapy shell "https://quotes.toscrape.com/page/1/" You will see something like: [ ... Scrapy log here ... ] 2016-09-19 12:09:27 [scrapy.core.engine] DEBUG: Crawled (200) log: 2016-09-19 18:57:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://quotes. ˓→toscrape.com/page/1/>

split("/")[-2] filename = 'quotes-%s.html' % page with open(filename, 'wb') as f: f.write(response.body) self.log('Saved file %s' % filename) As you can see, our Spider subclasses scrapy.Spider and defines some attributes will see something like: 2.3. Scrapy Tutorial 13 Scrapy Documentation, Release 1.3.3 [ ... Scrapy log here ... ] 2016-09-19 12:09:27 [scrapy.core.engine] DEBUG: Crawled (200) log: 2016-09-19 18:57:19 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes. ˓→toscrape.com/page/1/>

our spider. You can see a log line for each URL defined in start_urls. Because these URLs are the starting ones, they have no referrers, which is shown at the end of the log line, where it says (referer: containing arguments (ie. & character) will not work. This is what the shell looks like: [ ... Scrapy log here ... ] [s] Available Scrapy objects: [s] 2010-08-19 21:45:59-0300 [default] INFO: Spider closed each depth level Usage example: $ scrapy parse http://www.example.com/ -c parse_item [ ... scrapy log lines crawling example.com spider ... ] >>> STATUS DEPTH LEVEL 1 <<< # Scraped Items -----------

our spider. You can see a log line for each URL defined in start_urls. Because these URLs are the starting ones, they have no referrers, which is shown at the end of the log line, where it says (referer: org/Computers/Programming/Languages/Python/Books/ This is what the shell looks like: [ ... Scrapy log here ... ] [s] Available Scrapy objects: [s] 2010-08-19 21:45:59-0300 [default] INFO: Spider closed each depth level Usage example: $ scrapy parse http://www.example.com/ -c parse_item [ ... scrapy log lines crawling example.com spider ... ] >>> STATUS DEPTH LEVEL 1 <<< # Scraped Items -----------