that: pip install scrapy Note: The same non-Python dependencies can be used to install Scrapy in Debian Jessie (8.0) and above. macOS Building Scrapy’s dependencies requires the presence of a C compiler response): page = response.url.split("/")[-2] filename = f'quotes-{page}.html' with open(filename, 'wb') as f: f.write(response.body) self.log(f'Saved file {filename}') As you can see, our Spider subclasses def parse(self, response): page = response.url.split("/")[-2] filename = f'quotes-{page}.html' with open(filename, 'wb') as f: f.write(response.body) The parse() method will be called to handle each of

[https://stackoverflow.com/tags/scrapy]. Ask or search questions in the Scrapy subreddit [https://www.reddit.com/r/scrapy/]. Search for questions on the archives of the scrapy-users mailing list [https://groups.google that: pip install scrapy Note The same non-Python dependencies can be used to install Scrapy in Debian Jessie (8.0) and above. Mac OS X Building Scrapy’s dependencies requires the presence of a C compiler split("/")[-2] filename = 'quotes-%s.html' % page with open(filename, 'wb') as f: f.write(response.body) self.log('Saved file %s' % filename) As you can see, our Spider

that: pip install scrapy Note The same non-python dependencies can be used to install Scrapy in Debian Wheezy (7.0) and above. Mac OS X Building Scrapy’s dependencies requires the presence of a C compiler split("/")[-2] filename = 'quotes-%s.html' % page with open(filename, 'wb') as f: f.write(response.body) self.log('Saved file %s' % filename) As you can see, our Spider split("/")[-2] filename = 'quotes-%s.html' % page with open(filename, 'wb') as f: f.write(response.body) The parse() method will be called to handle each of the requests for those

that: pip install scrapy Note The same non-python dependencies can be used to install Scrapy in Debian Wheezy (7.0) and above. Mac OS X Building Scrapy’s dependencies requires the presence of a C compiler split("/")[-2] filename = 'quotes-%s.html' % page with open(filename, 'wb') as f: f.write(response.body) self.log('Saved file %s' % filename) As you can see, our Spider split("/")[-2] filename = 'quotes-%s.html' % page with open(filename, 'wb') as f: f.write(response.body) The parse() method will be called to handle each of the requests for those

that: pip install scrapy Note: The same non-Python dependencies can be used to install Scrapy in Debian Jessie (8.0) and above. 2.2. Installation guide 9 Scrapy Documentation, Release 1.6.0 Mac OS X Scrapy Documentation, Release 1.6.0 (continued from previous page) with open(filename, 'wb') as f: f.write(response.body) self.log('Saved file %s' % filename) As you can see, our Spider subclasses page = response.url.split("/")[-2] filename = 'quotes-%s.html' % page with open(filename, 'wb') as f: f.write(response.body) The parse() method will be called to handle each of the requests for those URLs

that: pip install scrapy Note: The same non-Python dependencies can be used to install Scrapy in Debian Jessie (8.0) and above. 2.2. Installation guide 9 Scrapy Documentation, Release 1.5.2 Mac OS X page = response.url.split("/")[-2] filename = 'quotes-%s.html' % page with open(filename, 'wb') as f: f.write(response.body) self.log('Saved file %s' % filename) As you can see, our Spider subclasses page = response.url.split("/")[-2] filename = 'quotes-%s.html' % page with open(filename, 'wb') as f: f.write(response.body) The parse() method will be called to handle each of the requests for those URLs

that: pip install scrapy Note: The same non-Python dependencies can be used to install Scrapy in Debian Jessie (8.0) and above. macOS Building Scrapy’s dependencies requires the presence of a C compiler filename = f'quotes-{page}.html' with open(filename, 'wb') as f: (continues on next page) 12 Chapter 2. First steps Scrapy Documentation, Release 2.6.3 (continued from previous page) f.write(response write(response.body) self.log(f'Saved file {filename}') As you can see, our Spider subclasses scrapy.Spider and defines some attributes and methods: • name: identifies the Spider. It must be unique within a project

that: pip install scrapy Note: The same non-Python dependencies can be used to install Scrapy in Debian Jessie (8.0) and above. macOS Building Scrapy’s dependencies requires the presence of a C compiler filename = f'quotes-{page}.html' with open(filename, 'wb') as f: (continues on next page) 12 Chapter 2. First steps Scrapy Documentation, Release 2.5.1 (continued from previous page) f.write(response write(response.body) self.log(f'Saved file {filename}') As you can see, our Spider subclasses scrapy.Spider and defines some attributes and methods: • name: identifies the Spider. It must be unique within a project

that: pip install Scrapy Note: The same non-python dependencies can be used to install Scrapy in Debian Wheezy (7.0) and above. Archlinux You can follow the generic instructions or install Scrapy from parse(self, response): filename = response.url.split("/")[-2] + '.html' with open(filename, 'wb') as f: f.write(response.body) Crawling To put our spider to work, go to the project’s top level directory as callback for parsing the response • --pipelines: process items through pipelines • --rules or -r: use CrawlSpider rules to discover the callback (i.e. spider method) to use for parsing the response

[https://stackoverflow.com/tags/scrapy]. Ask or search questions in the Scrapy subreddit [https://www.reddit.com/r/scrapy/]. Search for questions on the archives of the scrapy-users mailing list [https://groups.google that: pip install scrapy Note The same non-Python dependencies can be used to install Scrapy in Debian Jessie (8.0) and above. Mac OS X Building Scrapy’s dependencies requires the presence of a C compiler the suggested resources in the learnpython-subreddit [https://www.reddit.com/r/learnpython/wiki/index#wiki_new_to_python.3F]. Creating a project Before you start scraping, you will have to set up a new