Exporting Go Robert Griesemer GopherCon Singapore, 2017 Intro ● Go package ○ Namespace ○ Interface (export) ○ Import ● Implementation ○ Export/import (this talk) ○ Linker (not this talk) A long type struct field Factor field Power field Link basic int pointer Packages Objects Types Exporting means serializing a graph 7 2 5 6 3 1 4 1 2 3 4 5 #4 6 #2 7 serial form Algorithm: - Traverse

Built-in support for selecting and extracting data from HTML and XML sources • Built-in support for exporting data in multiple formats, including XML, CSV and JSON • A media pipeline for automatically downloading Documentation, Release 0.9 File Export Pipeline settings EXPORT_FORMAT The format to use for exporting. Here is a list of all available formats. Click on the respective Item Exporter to get more info export_empty_fields Item Exporter attribute. EXPORT_ENCODING Default: ’utf-8’ The encoding to use for exporting. Ths will be used for the encoding Item Exporter attribute. Items Define the data you want to scrape

Built-in support for selecting and extracting data from HTML and XML sources Built-in support for exporting data in multiple formats, including XML, CSV and JSON A media pipeline for automatically downloading ['name', 'price', 'description'] File Export Pipeline settings EXPORT_FORMAT The format to use for exporting. Here is a list of all available formats. Click on the respective Item Exporter to get more info export_empty_fields Item Exporter attribute. EXPORT_ENCODING Default: 'utf-8' The encoding to use for exporting. Ths will be used for the encoding Item Exporter attribute. © Copyright 2008-2010, Insophia. Last

method start_exporting() in order to signal the beginning of the exporting process 2. call the export_item() method for each item you want to export 3. and finally call the finish_exporting() to signal signal the end of the exporting process Here you can see an Item Pipeline which uses an Item Exporter to export scraped items to different files, one per spider: from scrapy.xlib.pydispatch import dispatcher exporter = XmlItemExporter(file) self.exporter.start_exporting() def spider_closed(self, spider): self.exporter.finish_exporting() file = self.files.pop(spider) file.close()

method start_exporting() in order to signal the beginning of the exporting process 2. call the export_item() method for each item you want to export 3. and finally call the finish_exporting() to signal signal the end of the exporting process Here you can see an Item Pipeline which uses an Item Exporter to export scraped items to different files, one per spider: from scrapy.xlib.pydispatch import dispatcher file self.exporter = XmlItemExporter(file) self.exporter.start_exporting() def spider_closed(self, spider): self.exporter.finish_exporting() file = self.files.pop(spider) file.close() def process_item(self

method start_exporting() in order to signal the beginning of the exporting process 2. call the export_item() method for each item you want to export 3. and finally call the finish_exporting() to signal signal the end of the exporting process Here you can see an Item Pipeline which uses an Item Exporter to export scraped items to different files, one per spider: from scrapy.xlib.pydispatch import dispatcher exporter = XmlItemExporter(file) self.exporter.start_exporting() def spider_closed(self, spider): self.exporter.finish_exporting() file = self.files.pop(spider) file.close()

method start_exporting() in order to signal the beginning of the exporting process 2. call the export_item() method for each item you want to export 3. and finally call the finish_exporting() to signal signal the end of the exporting process Here you can see an Item Pipeline which uses an Item Exporter to export scraped items to different files, one per spider: from scrapy.xlib.pydispatch import dispatcher file self.exporter = XmlItemExporter(file) self.exporter.start_exporting() def spider_closed(self, spider): self.exporter.finish_exporting() file = self.files.pop(spider) file.close() def process_item(self

method start_exporting() in order to signal the beginning of the exporting process 2. call the export_item() method for each item you want to export 3. and finally call the finish_exporting() to signal signal the end of the exporting process Here you can see an Item Pipeline which uses an Item Exporter to export scraped items to different files, one per spider: from scrapy import signals from scrapy.contrib file self.exporter = XmlItemExporter(file) self.exporter.start_exporting() def spider_closed(self, spider): self.exporter.finish_exporting() file = self.files.pop(spider) file.close() def process_item(self

have instantiated you exporter, you have to: 1. call the method start_exporting() in order to signal the beginning of the exporting process 7.5. Item Exporters 161 Scrapy Documentation, Release 0.18.4 export_item() method for each item you want to export 3. and finally call the finish_exporting() to signal the end of the exporting process Here you can see an Item Pipeline which uses an Item Exporter to export file self.exporter = XmlItemExporter(file) self.exporter.start_exporting() def spider_closed(self, spider): self.exporter.finish_exporting() file = self.files.pop(spider) file.close() def process_item(self

method start_exporting() in order to signal the beginning of the exporting process 2. call the export_item() method for each item you want to export 3. and finally call the finish_exporting() to signal signal the end of the exporting process Here you can see an Item Pipeline which uses an Item Exporter to export scraped items to different files, one per spider: from scrapy import signals from scrapy.contrib exporter = XmlItemExporter(file) self.exporter.start_exporting() def spider_closed(self, spider): self.exporter.finish_exporting() file = self.files.pop(spider) file.close()