. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 4.5 Web Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136 s3://mybucket/path/to/export.csv – s3://aws_key:aws_secret@mybucket/path/to/export.csv • Required external libraries: botocore (Python 2 and Python 3) or boto (Python 2 only) The AWS credentials can be passed as as user/password in the URI, or they can be passed through the following settings: • AWS_ACCESS_KEY_ID • AWS_SECRET_ACCESS_KEY Standard output The feeds are written to the standard output of the Scrapy

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 4.5 Web Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150 stored on Amazon S3. • URI scheme: s3 • Example URIs: – s3://mybucket/path/to/export.csv – s3://aws_key:aws_secret@mybucket/path/to/export.csv • Required external libraries: botocore (Python 2 and Python Documentation, Release 1.8.4 The AWS credentials can be passed as user/password in the URI, or they can be passed through the following settings: • AWS_ACCESS_KEY_ID • AWS_SECRET_ACCESS_KEY You can also

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 4.5 Web Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 stored on Amazon S3. • URI scheme: s3 • Example URIs: – s3://mybucket/path/to/export.csv – s3://aws_key:aws_secret@mybucket/path/to/export.csv 3.8. Feed exports 85 Scrapy Documentation, Release 1.7.4 3) or boto (Python 2 only) The AWS credentials can be passed as user/password in the URI, or they can be passed through the following settings: • AWS_ACCESS_KEY_ID • AWS_SECRET_ACCESS_KEY You can also

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 4.5 Web Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 URIs: – s3://mybucket/path/to/export.csv – s3://aws_key:aws_secret@mybucket/path/to/export.csv • Required external libraries: botocore or boto The AWS credentials can be passed as user/password in the following settings: 78 Chapter 3. Basic concepts Scrapy Documentation, Release 1.3.3 • AWS_ACCESS_KEY_ID • AWS_SECRET_ACCESS_KEY Standard output The feeds are written to the standard output of the

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 4.5 Web Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 Example URIs: – s3://mybucket/path/to/export.csv – s3://aws_key:aws_secret@mybucket/path/to/export.csv • Required external libraries: botocore The AWS credentials can be passed as user/password in the URI URI, or they can be passed through the following settings: • AWS_ACCESS_KEY_ID • AWS_SECRET_ACCESS_KEY You can also define a custom ACL for exported feeds using this setting: • FEED_STORAGE_S3_ACL

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150 4.5 Web Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 Example URIs: – s3://mybucket/path/to/export.csv – s3://aws_key:aws_secret@mybucket/path/to/export.csv • Required external libraries: botocore The AWS credentials can be passed as user/password in the URI URI, or they can be passed through the following settings: • AWS_ACCESS_KEY_ID • AWS_SECRET_ACCESS_KEY You can also define a custom ACL for exported feeds using this setting: • FEED_STORAGE_S3_ACL

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 4.5 Web Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 URIs: – s3://mybucket/path/to/export.csv – s3://aws_key:aws_secret@mybucket/path/to/export.csv • Required external libraries: botocore >= 1.4.87 The AWS credentials can be passed as user/password in the the URI, or they can be passed through the following settings: • AWS_ACCESS_KEY_ID • AWS_SECRET_ACCESS_KEY You can also define a custom ACL for exported feeds using this setting: • FEED_STORAGE_S3_ACL

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 4.5 Web Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 Example URIs: – s3://mybucket/path/to/export.csv – s3://aws_key:aws_secret@mybucket/path/to/export.csv • Required external libraries: botocore The AWS credentials can be passed as user/password in the URI URI, or they can be passed through the following settings: • AWS_ACCESS_KEY_ID • AWS_SECRET_ACCESS_KEY You can also define a custom ACL for exported feeds using this setting: • FEED_STORAGE_S3_ACL

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 4.5 Web Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 Example URIs: – s3://mybucket/path/to/export.csv – s3://aws_key:aws_secret@mybucket/path/to/export.csv • Required external libraries: botocore The AWS credentials can be passed as user/password in the URI URI, or they can be passed through the following settings: • AWS_ACCESS_KEY_ID • AWS_SECRET_ACCESS_KEY You can also define a custom ACL for exported feeds using this setting: • FEED_STORAGE_S3_ACL

Console Inspect a running crawler using a built-in Python console. Web Service Monitor and control a crawler using a web service. Solving specific problems Frequently Asked Questions Get answers to change the export format (XML or CSV, for example) or the storage backend (FTP or Amazon S3 [https://aws.amazon.com/s3/], for example). You can also write an item pipeline to store the items in a database none S3 The feeds are stored on Amazon S3 [https://aws.amazon.com/s3/]. URI scheme: s3 Example URIs: s3://mybucket/path/to/export.csv s3://aws_key:aws_secret@mybucket/path/to/export.csv Required external