html' with open(filename, 'wb') as f: f.write(response.body) Crawling To put our spider to work, go to the project’s top level directory and run: scrapy crawl dmoz This command runs the spider with through examples, and this tutorial to learn “how to think in XPath”. Note: CSS vs XPath: you can go a long way extracting data from web pages using only CSS selectors. However, XPath offers more power your system. 2.3. Scrapy Tutorial 13 Scrapy Documentation, Release 1.0.7 To start a shell, you must go to the project’s top level directory and run: scrapy shell "http://www.dmoz.org/Computers/Program

split("/")[-2] with open(filename, 'wb') as f: f.write(response.body) Crawling To put our spider to work, go to the project’s top level directory and run: scrapy crawl dmoz The crawl dmoz command runs the spider requires IPython (an extended Python console) installed on your system. To start a shell, you must go to the project’s top level directory and run: scrapy shell "http://www.dmoz.org/Computers/Program startproject myproject That will create a Scrapy project under the myproject directory. Next, you go inside the new project directory: cd myproject And you’re ready to use the scrapy command to manage

open(filename, 'wb') as f: f.write(response.body) Crawling To put our spider to work, go to the project’s top level directory and run: scrapy crawl dmoz This command runs the spider with learn “how to think in XPath” [http://plasmasturm.org/log/xpath101/]. Note CSS vs XPath: you can go a long way extracting data from web pages using only CSS selectors. However, XPath offers more power [http://ipython.org/] (an extended Python console) installed on your system. To start a shell, you must go to the project’s top level directory and run: scrapy shell "http://www.dmoz.org/Computers/Progra

follow and creating new requests (Request) from them. How to run our spider To put our spider to work, go to the project’s top level directory and run: scrapy crawl quotes This command runs the spider with project_dir directory. If project_dir wasn’t specified, project_dir will be the same as myproject. Next, you go inside the new project directory: cd project_dir And you’re ready to use the scrapy command to manage href="#">Click here to go to the Next Page') Converting a node-set to string: >>> sel.xpath('//a//text()').getall() # take a peek at the node-set ['Click here to go to the ', 'Next