Tabular data (e.g. data stored in CSV and Excel worksheets) is one of the most common forms of data available on the web. This guide will walk through validating tabular data using Frictionless Data software.
This guide show how you can validate your tabular data and check both:
- Structure: are there too many rows or columns in some places?
- Schema: does the data fit its schema. Are the values in the date column actually dates? Are all the numbers greater than zero?
We will walk through two methods of performing validation:
- Web service: an online service called goodtables. This option requires no technical knowledge or expertise.
- Using the Python goodtables library. This allows you full control over the validation process but requires knowledge of Python.
goodtables is a free, open-source, hosted service for validating tabular data. goodtables checks your data for its structure, and, optionally, its adherence to a specified schema. goodtables will give quick and simple feedback on where your tabular data may not yet be quite perfect.
To start, all you need to do is upload or provide a link to a CSV file and hit the "Validate" button.
If your data is structurally valid, you should receive the following result:
The report should highlight the structural issues found in your data for correction. For instance, a poorly structured tabular dataset may consist of a header row with too many (or too few) columns when compared to of data rows with an equal amount of columns.
You can also provide a schema for your tabular data defined using JSON Table Schema.
Briefly, the format allows users to specify not only the types of information within each column in a tabular dataset, but also expected values. For more information, see the Table Schema guide or the full standard.
Python + goodtables
goodtables is also available as a Python library. The following short
snippets demonstrate examples of loading and validating data in a file
data.csv(and in the second example, validating the same data file against
from goodtables import validate report = validate('data.csv') report['valid'] report['table-count'] report['error-count'] report['tables']['valid'] report['tables']['source'] report['tables']['errors']['code']
from goodtables import validate # sync source/schema fields order report = validate('data.csv', schema='schema.json', order_fields=True) ...
Continuous Data Validation
In a bid to streamline the process of data validation and ensure seamless integration is possible in different publishing workflows, we have set up a continuous data validation hosted service that builds on top of Frictionless Data libraries. goodtables.io provides support for different backends. At this time, users can use it to check any datasets hosted on GitHub and Amazon S3 buckets, automatically running validation against data files every time they are updated, and providing a user friendly report of any issues found.
Start your continuous data validation here: https://goodtables.io
Blog post on goodtables python library and goodtables web service: http://okfnlabs.org/blog/2017/05/22/introducing-the-new-goodtables-library-and-goodtablesio.html
README.md for more information.