4 Chapter 1. Getting help CHAPTER 2 First steps Scrapy at a glance Scrapy is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful multi-purpose helper for dealing with URLs and web page encodings • twisted, an asynchronous networking framework • cryptography and pyOpenSSL, to deal with various network-level security needs The minimal versions is used instead to create the Requests. This method is also called only once from Scrapy, so it’s safe to implement it as a generator. The default implementation uses make_requests_from_url() to generate

4 Chapter 1. Getting help CHAPTER 2 First steps Scrapy at a glance Scrapy is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful multi-purpose helper for dealing with URLs and web page encodings • twisted, an asynchronous networking framework • cryptography and pyOpenSSL, to deal with various network-level security needs The minimal versions is used instead to create the Requests. This method is also called only once from Scrapy, so it’s safe to implement it as a generator. The default implementation uses make_requests_from_url() to generate

Chapter 1. Getting help CHAPTER 2 First steps 2.1 Scrapy at a glance Scrapy is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful multi-purpose helper for dealing with URLs and web page encodings • twisted, an asynchronous networking framework • cryptography and pyOpenSSL, to deal with various network-level security needs The minimal versions It is called by Scrapy when the spider is opened for scraping. Scrapy calls it only once, so it is safe to implement start_requests() as a generator. The default implementation generates Request(url, dont_filter=True)

Chapter 1. Getting help CHAPTER 2 First steps 2.1 Scrapy at a glance Scrapy is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful multi-purpose helper for dealing with URLs and web page encodings • twisted, an asynchronous networking framework • cryptography and pyOpenSSL, to deal with various network-level security needs The minimal versions It is called by Scrapy when the spider is opened for scraping. Scrapy calls it only once, so it is safe to implement start_requests() as a generator. The default implementation generates Request(url, dont_filter=True)

Understand Scrapy versioning and API stability. Scrapy at a glance Scrapy is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful with URLs and web page encodings twisted [https://twistedmatrix.com/], an asynchronous networking framework cryptography [https://cryptography.io/] and pyOpenSSL [https://pypi.python.org/pypi/pyOpenSSL] is used instead to create the Requests. This method is also called only once from Scrapy, so it’s safe to implement it as a generator. The default implementation uses make_requests_from_url() to generate

Understand Scrapy versioning and API stability. Scrapy at a glance Scrapy is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful with URLs and web page encodings twisted [https://twistedmatrix.com/], an asynchronous networking framework cryptography [https://cryptography.io/] and pyOpenSSL [https://pypi.python.org/pypi/pyOpenSSL] is used instead to create the Requests. This method is also called only once from Scrapy, so it’s safe to implement it as a generator. The default implementation uses make_requests_from_url() to generate

4 Chapter 1. Getting help CHAPTER 2 First steps Scrapy at a glance Scrapy is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful multi-purpose helper for dealing with URLs and web page encodings • twisted, an asynchronous networking framework • cryptography and pyOpenSSL, to deal with various network-level security needs The minimal versions It is called by Scrapy when the spider is opened for scraping. Scrapy calls it only once, so it is safe to implement start_requests() as a generator. The default implementation generates Request(url, dont_filter=True)

Scrapy Documentation, Release 1.7.4 Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide Chapter 1. Getting help CHAPTER 2 First steps 2.1 Scrapy at a glance Scrapy is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful multi-purpose helper for dealing with URLs and web page encodings • twisted, an asynchronous networking framework • cryptography and pyOpenSSL, to deal with various network-level security needs The minimal versions

Scrapy Documentation, Release 1.8.4 Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide Chapter 1. Getting help CHAPTER TWO FIRST STEPS 2.1 Scrapy at a glance Scrapy is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful multi-purpose helper for dealing with URLs and web page encodings • twisted, an asynchronous networking framework • cryptography and pyOpenSSL, to deal with various network-level security needs The minimal versions

Chapter 1. Getting help CHAPTER 2 First steps 2.1 Scrapy at a glance Scrapy is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful is used instead to create the Requests. This method is also called only once from Scrapy, so it’s safe to implement it as a generator. The default implementation uses make_requests_from_url() to generate it’s intended to perform any last time processing required before returning the results to the framework core, for example setting the item IDs. It receives a list of results and the response which originated