httpbin.org/status/500", # server issue "http://www.httpbin.org:12345/", # non-responding host, timeout ˓→expected "http://www.httphttpbinbin.org/", # DNS error expected ] def start_requests(self): dont_obey_robotstxt 3.9. Requests and Responses 95 Scrapy Documentation, Release 2.0.1 • download_timeout • download_maxsize • download_latency • download_fail_on_dataloss • proxy • ftp_user (See FTP_USER use for the performing the request. download_timeout The amount of time (in secs) that the downloader will wait before timing out. See also: DOWNLOAD_TIMEOUT. download_latency The amount of time spent

httpbin.org/status/500", # server issue "http://www.httpbin.org:12345/", # non-responding host, timeout ˓→expected "http://www.httphttpbinbin.org/", # DNS error expected ] def start_requests(self): dont_cache • redirect_reasons • redirect_urls • bindaddress • dont_obey_robotstxt • download_timeout • download_maxsize • download_latency • download_fail_on_dataloss • proxy • ftp_user (See FTP_USER use for the performing the request. download_timeout The amount of time (in secs) that the downloader will wait before timing out. See also: DOWNLOAD_TIMEOUT. 96 Chapter 3. Basic concepts Scrapy Documentation

httpbin.org/status/500", # server issue "http://www.httpbin.org:12345/", # non-responding host, timeout ˓→expected "http://www.httphttpbinbin.org/", # DNS error expected ] def start_requests(self): dont_cache • redirect_reasons • redirect_urls • bindaddress • dont_obey_robotstxt • download_timeout • download_maxsize • download_latency • download_fail_on_dataloss • proxy • ftp_user (See FTP_USER use for the performing the request. download_timeout The amount of time (in secs) that the downloader will wait before timing out. See also: DOWNLOAD_TIMEOUT. download_latency The amount of time spent

httpbin.org/status/500", # server issue "http://www.httpbin.org:12345/", # non-responding host, timeout ˓→expected "http://www.httphttpbinbin.org/", # DNS error expected ] def start_requests(self): dont_cache • redirect_reasons • redirect_urls • bindaddress • dont_obey_robotstxt • download_timeout • download_maxsize • download_latency • download_fail_on_dataloss • proxy • ftp_user (See FTP_USER use for the performing the request. download_timeout The amount of time (in secs) that the downloader will wait before timing out. See also: DOWNLOAD_TIMEOUT. download_latency The amount of time spent

httpbin.org/status/500", # server issue "http://www.httpbin.org:12345/", # non-responding host, timeout ˓→expected "http://www.httphttpbinbin.org/", # DNS error expected ] def start_requests(self): dont_cache • redirect_reasons • redirect_urls • bindaddress • dont_obey_robotstxt • download_timeout • download_maxsize • download_latency • download_fail_on_dataloss • proxy • ftp_user (See FTP_USER use for the performing the request. download_timeout The amount of time (in secs) that the downloader will wait before timing out. See also: DOWNLOAD_TIMEOUT. download_latency The amount of time spent

httpbin.org/status/500", # server issue "http://www.httpbin.org:12345/", # non-responding host, timeout ˓→expected "http://www.httphttpbinbin.org/", # DNS error expected ] def start_requests(self): dont_obey_robotstxt • download_timeout • download_maxsize • download_latency • proxy bindaddress The IP of the outgoing IP address to use for the performing the request. download_timeout The amount of time time (in secs) that the downloader will wait before timing out. See also: DOWNLOAD_TIMEOUT. download_latency The amount of time spent to fetch the response, since the request has been started, i.e. HTTP

httpbin.org/status/500", # server issue "http://www.httpbin.org:12345/", # non-responding host, timeout expected "https://example.invalid/", # DNS error expected ] def start_requests(self): for u in dont_redirect • dont_retry • download_fail_on_dataloss • download_latency • download_maxsize • download_timeout • ftp_password (See FTP_PASSWORD for more info) • ftp_user (See FTP_USER for more info) • handle_httpstatus_all use for the performing the request. download_timeout The amount of time (in secs) that the downloader will wait before timing out. See also: DOWNLOAD_TIMEOUT. download_latency The amount of time spent

dont_obey_robotstxt • download_timeout • download_maxsize • proxy bindaddress The IP of the outgoing IP address to use for the performing the request. download_timeout The amount of time (in secs) secs) that the downloader will wait before timing out. See also: DOWNLOAD_TIMEOUT. Request subclasses Here is the list of built-in Request subclasses. You can also subclass it to implement your own custom enable DNS in-memory cache. DNSCACHE_SIZE Default: 10000 DNS in-memory cache size. DNS_TIMEOUT Default: 60 Timeout for processing of DNS queries in seconds. Float is supported. DOWNLOADER Default: 'scrapy

httpbin.org/status/500", # server issue "http://www.httpbin.org:12345/", # non-responding host, timeout ˓→expected "http://www.httphttpbinbin.org/", # DNS error expected ] def start_requests(self): dont_obey_robotstxt • download_timeout • download_maxsize • download_latency • proxy bindaddress The IP of the outgoing IP address to use for the performing the request. download_timeout The amount of time time (in secs) that the downloader will wait before timing out. See also: DOWNLOAD_TIMEOUT. download_latency The amount of time spent to fetch the response, since the request has been started, i.e. HTTP

httpbin.org/status/500", # server issue "http://www.httpbin.org:12345/", # non-responding host, timeout ˓→expected "http://www.httphttpbinbin.org/", # DNS error expected ] def start_requests(self): dont_obey_robotstxt • download_timeout • download_maxsize • proxy bindaddress The IP of the outgoing IP address to use for the performing the request. download_timeout The amount of time (in secs) secs) that the downloader will wait before timing out. See also: DOWNLOAD_TIMEOUT. Request subclasses Here is the list of built-in Request subclasses. You can also subclass it to implement your own custom