. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 3.11 Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . fault-tolerant way) Scrapy also gives you control over the politeness of the crawl through a few settings. You can do things like setting a download delay between each request, limiting amount of concurrent will create a tutorial directory with the following contents: tutorial/ scrapy.cfg # deploy configuration file tutorial/ # project's Python module, you'll import your code from here __init__.py items

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 3.11 Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . fault-tolerant way) Scrapy also gives you control over the politeness of the crawl through a few settings. You can do things like setting a download delay between each request, limiting amount of concurrent will create a tutorial directory with the following contents: tutorial/ scrapy.cfg # deploy configuration file tutorial/ # project's Python module, you'll import your code from here __init__.py items

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 3.11 Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . fault-tolerant way) Scrapy also gives you control over the politeness of the crawl through a few settings. You can do things like setting a download delay between each request, limiting amount of concurrent will create a tutorial directory with the following contents: tutorial/ scrapy.cfg # deploy configuration file tutorial/ # project's Python module, you'll import your code from here __init__.py items

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 3.11 Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . fault-tolerant way) Scrapy also gives you control over the politeness of the crawl through a few settings. You can do things like setting a download delay between each request, limiting amount of concurrent will create a tutorial directory with the following contents: tutorial/ scrapy.cfg # deploy configuration file tutorial/ # project's Python module, you'll import your code from here __init__.py items

Extractors Convenient classes to extract links to follow from pages. Settings Learn how to configure Scrapy and see all available settings. Exceptions See all available exceptions and their meaning. Built-in fault-tolerant way) Scrapy also gives you control over the politeness of the crawl through a few settings. You can do things like setting a download delay between each request, limiting amount of concurrent tutorial directory with the following contents: tutorial/ scrapy.cfg # deploy configuration file tutorial/ # project's Python module, you'll import your code from here

Extractors Convenient classes to extract links to follow from pages. Settings Learn how to configure Scrapy and see all available settings. Exceptions See all available exceptions and their meaning. Built-in fault-tolerant way) Scrapy also gives you control over the politeness of the crawl through a few settings. You can do things like setting a download delay between each request, limiting amount of concurrent tutorial directory with the following contents: tutorial/ scrapy.cfg # deploy configuration file tutorial/ # project's Python module, you'll import your code from here

Extractors Convenient classes to extract links to follow from pages. Settings Learn how to configure Scrapy and see all available settings. Exceptions See all available exceptions and their meaning. Built-in fault-tolerant way) Scrapy also gives you control over the politeness of the crawl through a few settings. You can do things like setting a download delay between each request, limiting amount of concurrent tutorial directory with the following contents: tutorial/ scrapy.cfg # deploy configuration file tutorial/ # project's Python module, you'll import your code from here

Responses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 7.2 Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . scrapy.cfg tutorial/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ... These are basically: • scrapy.cfg: the project configuration file • tutorial/: the project’s python module, you’ll project’s items file. • tutorial/pipelines.py: the project’s pipelines file. • tutorial/settings.py: the project’s settings file. 2.3. Scrapy Tutorial 11 Scrapy Documentation, Release 0.14.4 • tutorial/spiders/:

Responses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 7.2 Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . scrapy.cfg dmoz/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ... These are basically: • scrapy.cfg: the project configuration file • dmoz/: the project’s python module, you’ll project’s items file. • dmoz/pipelines.py: the project’s pipelines file. • dmoz/settings.py: the project’s settings file. • dmoz/spiders/: a directory where you’ll later put your spiders. 2.3. Scrapy

Understand the classes used to represent HTTP requests and responses. Settings Learn how to configure Scrapy and see all available settings. Signals See all available signals and how to work with them. items.py pipelines.py settings.py spiders/ __init__.py ... These are basically: scrapy.cfg: the project configuration file tutorial/: the project’s python project’s items file. tutorial/pipelines.py: the project’s pipelines file. tutorial/settings.py: the project’s settings file. tutorial/spiders/: a directory where you’ll later put your spiders. Defining