Granicus Open Data comes standard with a Datastore to house tabular data imported from your CSV files on Granicus Open Data. That is, the Datastore can support files with contents that appear as a table (rows and columns). You can think of the Datastore like a basic database. Files that are imported to the Datastore have the contents of the file copied into a table in the Datastore, and the Datastore as a whole is composed of all the tables copied from imported files on Granicus Open Data.The Datastore processes data, stores the contents of Resources (if CSV), and makes them ready to be queried.
As a Site Manager, you can manage the Datastore by adding and removing files from the Datastore. In most cases you want all CSV files included in the Datastore to support better data previewing, large files, and a more robust API.
Managing the Datastore.
In broad strokes, managing the Granicus Open Data Datastore is deciding which Resources to include in the Datastore. There isn't any management further than that, and every user has the ability to import and remove Resources they've authored. As a Site Manager, you can import or remove any Resource regardless of the author. This allows you to manage what data is included in the Datastore API.
There may be some sensitive data that should not be included in the Datastore, but in general we recommend increasing your transparency and usability of your data by importing every Resource possible into your Datastore.
Importing and removing files.
Uploading files to the Datastore has major benefits including enhancing the Datastore API and improved user experience of previewing data. The Datastore API makes the Resources more usable and accessible to technical users. Previews display resources as graphs, grids, or maps for geospatial data. In some cases files contain thousands (or millions) of rows. For data on such high order, users can only properly preview the data if the Resource has been imported into the Granicus Open Data Datastore.
When to import:
The file is formatted as a CSV.
The file is very large (and formatted as a CSV).
To include the file in the Datastore API.
Import a file into the Datastore. Once a Resource is added, you can import the file into the Datastore. Click the Manage Datastore button at the top of the screen to open the options included for importing a file into the Datastore. From the import screen, you can select the options that best reflect the contents of your data. By default, Granicus Open Data is set to match common data formatting practices.
In the example below, the Site Manager is importing a Resource into the Datastore from the Manage Datastore page. From the Import tab, the Site Manager selects the delimeter and leaves the rest of the default settings for import. After clicking the Import button at the bottom of the page, the contents of the Resource are now included in the Granicus Open Data Datastore as well as the Datastore API.
Removing a file from the Datastore. If you need to remove a Resource from the Datastore you’ll make this change directly from editing the Resource on the Manage Datastore screen. You have two options for removing the contents from the Datastore:
Delete items. By deleting items, the values are removed from the Datastore. However, the table itself generated by the file import still exists. In this sense, you can think of the file as imported to the Datastore but empty. Again, the CSV file on your Granicus Open Data site is separate from the Datastore. By deleting the values in the Datastore, your file won’t be changed.
- Drop Datastore. This option removes both the values in the file as well as the table generated upon import. In this sense, the file is not imported. If you don’t want the contents of a file in the Datastore, we recommend this option.
Updating Resources in the Datastore. When you import a Resource to the Datastore, the file contents are copied to the Datastore and they exist separately from your Granicus Open Data site. If you update your Resource on your Granicus Open Data site, you will also need to update the file in the Datastore. You can simply import the file again, and the Datastore will overwrite the existing contents to reflect the changes. You’ll see the changes in the number of items under Datastore File: Status item.
Using Fast Import
For datasets with thousands (or millions) of rows, the size of the file increases quickly. By default, DKAN processes the contents of a file before importing to the Datastore. The processing checks that the contents match delimiter, file encoding, etc. for a smooth import.
Large file sizes can delay the time it takes to import a file into the Datastore as more information needs to be processed before it can be imported.
With the Fast Import option, the time to upload a large file is dramatically reduced. The contents of a file are not processed and directly imported to the Datastore, so files that otherwise might take hours to import only take minutes.
For smaller files, we recommend using the standard processor and import. When large files must be imported, check the Use Fast Import checkbox to skip the processing and directly import.
Special note: This capability is not enabled by default. Talk with your GovDelivery implementation consultant for more details on set up.