Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 5.2 Debugging Spiders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . scraping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 5.8 Debugging memory leaks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 they get broken • An Interactive shell console for trying XPaths, very useful for writing and debugging your spiders • A System service designed to ease the deployment and run of your spiders in production

Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 5.2 Debugging Spiders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . scraping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 5.8 Debugging memory leaks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 they get broken • An Interactive shell console for trying XPaths, very useful for writing and debugging your spiders • A System service designed to ease the deployment and run of your spiders in production

Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 5.2 Debugging Spiders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . scraping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 5.8 Debugging memory leaks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 they get broken • An Interactive shell console for trying XPaths, very useful for writing and debugging your spiders • A System service designed to ease the deployment and run of your spiders in production

Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 5.2 Debugging Spiders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . scraping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 5.8 Debugging memory leaks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 they get broken • An Interactive shell console for trying XPaths, very useful for writing and debugging your spiders • A System service designed to ease the deployment and run of your spiders in production

scraping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 5.4 Debugging memory leaks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 crawl depth restriction, etc • An Interactive scraping shell console, very useful for writing and debugging your spiders • A builtin Web service for monitoring and controlling your bot • A Telnet console get familiarized with the Scrapy shell you’ll see that it’s an invaluable tool for developing and debugging your spiders. If you have IPython installed, the Scrapy shell will use it (instead of the standard

Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 5.2 Debugging Spiders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . scraping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 5.8 Debugging memory leaks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 they get broken • An Interactive shell console for trying XPaths, very useful for writing and debugging your spiders • A System service designed to ease the deployment and run of your spiders in production

some useful add-ons. Using Firebug for scraping Learn how to scrape efficiently using Firebug. Debugging memory leaks Learn how to find and get rid of memory leaks in your crawler. Downloading Item Images crawl depth restriction, etc An Interactive scraping shell console, very useful for writing and debugging your spiders A builtin Web service for monitoring and controlling your bot A Telnet console for get familiarized with the Scrapy shell you’ll see that it’s an invaluable tool for developing and debugging your spiders. If you have IPython [http://ipython.scipy.org/] installed, the Scrapy shell will

specific problems Frequently Asked Questions Get answers to most frequently asked questions. Debugging Spiders Learn how to debug common problems of your scrapy spider. Spiders Contracts Learn how some useful add-ons. Using Firebug for scraping Learn how to scrape efficiently using Firebug. Debugging memory leaks Learn how to find and get rid of memory leaks in your crawler. Downloading Item Images when they get broken An Interactive shell console for trying XPaths, very useful for writing and debugging your spiders A System service designed to ease the deployment and run of your spiders in production

specific problems Frequently Asked Questions Get answers to most frequently asked questions. Debugging Spiders Learn how to debug common problems of your scrapy spider. Spiders Contracts Learn how some useful add-ons. Using Firebug for scraping Learn how to scrape efficiently using Firebug. Debugging memory leaks Learn how to find and get rid of memory leaks in your crawler. Downloading Item Images when they get broken An Interactive shell console for trying XPaths, very useful for writing and debugging your spiders A System service designed to ease the deployment and run of your spiders in production

specific problems Frequently Asked Questions Get answers to most frequently asked questions. Debugging Spiders Learn how to debug common problems of your scrapy spider. Spiders Contracts Learn how some useful add-ons. Using Firebug for scraping Learn how to scrape efficiently using Firebug. Debugging memory leaks Learn how to find and get rid of memory leaks in your crawler. Downloading Item Images when they get broken An Interactive shell console for trying XPaths, very useful for writing and debugging your spiders A System service designed to ease the deployment and run of your spiders in production