Tabular data (e.g. data stored in CSV and Excel worksheets) is one of the most common forms of data available on the web. This guide will walk through validating tabular data using Frictionless Data software.
This guide show how you can validate your tabular data and check both:
- Structure: are there too many rows or columns in some places?
- Schema: does the data fit its schema. Are the values in the date column actually dates? Are all the numbers greater than zero?
We will walk through two methods of performing validation:
- Web service: an online service called goodtables. This option requires no technical knowledge or expertise.
- Using the Python goodtables library. This allows you full control over the validation process but requires knowledge of Python.
goodtables is a free, open-source, hosted
service for validating tabular data. goodtables checks your data for
its structure, and, optionally, its adherence to a specified schema. Where the latter fails, goodtables highlights content errors so you can fix them speedily.
goodtables will give quick and simple feedback on where your tabular
data may not yet be quite perfect.
To get started with one-off validation of your tabular datasets, use try.goodtables.io. All you need to do is upload or provide a link to a CSV
file and hit the “Validate” button.
If your data is structurally valid, you should receive the following
The report should highlight the structural issues found in your data
for correction. For instance, a poorly structured tabular dataset may
consist of a header row with too many (or too few) columns when
compared to of data rows with an equal amount of columns.
You can also provide a schema for your tabular data defined using JSON
Briefly, the format allows users to specify not only
the types of information within each column in a tabular dataset, but
also expected values. For more information, see the
Table Schema guide or
the full standard.
# Python + goodtables
goodtables is also available as a Python library. The following short
snippets demonstrate examples of loading and validating data in a file
data.csv(and in the second example, validating the same data file against
# Validating Structure
from goodtables import validate report = validate('data.csv') report['valid'] report['table-count'] report['error-count'] report['tables']['valid'] report['tables']['source'] report['tables']['errors']['code']
# Validating Schema
from goodtables import validate # sync source/schema fields order report = validate('data.csv', schema='schema.json', order_fields=True) ...
# Continuous Data Validation
In a bid to streamline the process of data validation and ensure seamless integration is possible in different publishing workflows, we have set up a continuous data validation hosted service that builds on top of Frictionless Data libraries. goodtables.io provides support for different backends. At this time, users can use it to check any datasets hosted on GitHub and Amazon S3 buckets, automatically running validation against data files every time they are updated, and providing a user friendly report of any issues found.
Start your continuous data validation here: https://goodtables.io
Blog post on goodtables python library and goodtables web service: http://okfnlabs.org/blog/2017/05/22/introducing-the-new-goodtables-library-and-goodtablesio.html
README.md for more information.
Find more examples on validating tabular data in the Frictionless Data Field Guide