item["screenshot_filename"] = filename return item Duplicates filter A filter that looks for duplicate items, and drops those items that were already processed. Let’s say that our items have a unique set() def process_item(self, item, spider): if item['id'] in self.ids_seen: raise DropItem("Duplicate item found: %s" % item) else: self.ids_seen.add(item['id']) return item 3.7.3 Activating an Item (using w3lib.url.canonicalize_url). Defaults to False. Note that canonicalize_url is meant for duplicate checking; it can change the URL visible at server side, so the response can be different for requests

item["screenshot_filename"] = filename return item Duplicates filter A filter that looks for duplicate items, and drops those items that were already processed. Let’s say that our items have a unique set() def process_item(self, item, spider): if item['id'] in self.ids_seen: raise DropItem("Duplicate item found: %s" % item) else: self.ids_seen.add(item['id']) return item 3.7.3 Activating an Item (using w3lib.url.canonicalize_url). Defaults to False. Note that canonicalize_url is meant for duplicate checking; it can change the URL visible at server side, so the response can be different for requests

item["screenshot_filename"] = filename return item Duplicates filter A filter that looks for duplicate items, and drops those items that were already processed. Let’s say that our items have a unique set() def process_item(self, item, spider): if item['id'] in self.ids_seen: raise DropItem("Duplicate item found: %s" % item) (continues on next page) 82 Chapter 3. Basic concepts Scrapy Documentation (using w3lib.url.canonicalize_url). Defaults to False. Note that canonicalize_url is meant for duplicate checking; it can change the URL visible at server side, so the response can be 3.10. Link Extractors

item["screenshot_filename"] = filename return item Duplicates filter A filter that looks for duplicate items, and drops those items that were already processed. Let’s say that our items have a unique set() def process_item(self, item, spider): if item['id'] in self.ids_seen: raise DropItem("Duplicate item found: %s" % item) else: self.ids_seen.add(item['id']) return item 88 Chapter 3. Basic concepts (using w3lib.url.canonicalize_url). Defaults to False. Note that canonicalize_url is meant for duplicate checking; it can change the URL visible at server side, so the response can be different for requests

adapter["screenshot_filename"] = filename return item Duplicates filter A filter that looks for duplicate items, and drops those items that were already processed. Let’s say that our items have a unique previous page) adapter = ItemAdapter(item) if adapter['id'] in self.ids_seen: raise DropItem("Duplicate item found: %r" % item) else: self.ids_seen.add(adapter['id']) return item 3.7.3 Activating an canonicalize_url is meant 3.10. Link Extractors 109 Scrapy Documentation, Release 2.2.1 for duplicate checking; it can change the URL visible at server side, so the response can be different for requests

adapter["screenshot_filename"] = filename return item Duplicates filter A filter that looks for duplicate items, and drops those items that were already processed. Let’s say that our items have a unique item, spider): adapter = ItemAdapter(item) if adapter['id'] in self.ids_seen: raise DropItem(f"Duplicate item found: {item!r}") else: self.ids_seen.add(adapter['id']) return item 3.7.3 Activating an (using w3lib.url.canonicalize_url). Defaults to False. Note that canonicalize_url is meant for duplicate checking; it can change the URL visible at server side, so the response can be 3.10. Link Extractors

adapter["screenshot_filename"] = filename return item Duplicates filter A filter that looks for duplicate items, and drops those items that were already processed. Let’s say that our items have a unique previous page) adapter = ItemAdapter(item) if adapter['id'] in self.ids_seen: raise DropItem("Duplicate item found: %r" % item) else: self.ids_seen.add(adapter['id']) return item 3.7.3 Activating an canonicalize_url is meant 110 Chapter 3. Basic concepts Scrapy Documentation, Release 2.3.0 for duplicate checking; it can change the URL visible at server side, so the response can be different for requests

item["screenshot_filename"] = filename return item Duplicates filter A filter that looks for duplicate items, and drops those items that were already processed. Let’s say that our items have a unique process_item(self, item, spider): if item['id'] in self.ids_seen: raise DropItem("Duplicate item found: %s" % item) else: self.ids_seen.add(item['id']) return (using w3lib.url.canonicalize_url). Defaults to False. Note that canonicalize_url is meant for duplicate checking; it can change the URL visible at server side, so the response can be different for requests

item["screenshot_filename"] = filename return item Duplicates filter A filter that looks for duplicate items, and drops those items that were already processed. Let’s say that our items have a unique set() def process_item(self, item, spider): if item['id'] in self.ids_seen: raise DropItem("Duplicate item found: %s" % item) else: self.ids_seen.add(item['id']) return item 3.7.3 Activating an Item (using w3lib.url.canonicalize_url). Defaults to False. Note that canonicalize_url is meant for duplicate checking; it can change the URL visible at server side, so the response can be different for requests

item["screenshot_filename"] = filename return item Duplicates filter A filter that looks for duplicate items, and drops those items that were already processed. Let’s say that our items have a unique set() def process_item(self, item, spider): if item['id'] in self.ids_seen: raise DropItem("Duplicate item found: %s" % item) else: self.ids_seen.add(item['id']) return item 3.7.3 Activating an Item canonicalize_url is meant 3.10. Link Extractors 105 Scrapy Documentation, Release 2.1.0 for duplicate checking; it can change the URL visible at server side, so the response can be different for requests