Singer Spec#

The Singer Specification is an open source standard for defining the format of data exchange. The standard is useful because it enables Data Professionals to move data between arbritrary systems as long as the programs generating and ingesting the data can understand this format.

This documentation is our attempt at simplifying the canonical specification into an easier to understand and follow format for people who are new to the Singer community. The full specification is in the Singer project on GitHub.

Version#

The current version of the spec is 0.3.0 and is versioned using Semantic Versioning.

Basics#

Messages#

The full specification for data exchange consists of three types of JSON-formatted messages: schema, record, and state. The record message contains the actual data being communicated, the schema message defines the structure of the data, and the state message keeps track of the progress of an extraction.

An example of what these messages look like is here:

{"type": "SCHEMA", "stream": "users", "key_properties": ["id"], "schema": {"required": ["id"], "type": "object", "properties": {"id": {"type": "integer"}}}}
{"type": "RECORD", "stream": "users", "record": {"id": 1, "name": "Chris"}}
{"type": "RECORD", "stream": "users", "record": {"id": 2, "name": "Mike"}}
{"type": "SCHEMA", "stream": "locations", "key_properties": ["id"], "schema": {"required": ["id"], "type": "object", "properties": {"id": {"type": "integer"}}}}
{"type": "RECORD", "stream": "locations", "record": {"id": 1, "name": "Philadelphia"}}
{"type": "STATE", "value": {"users": 2, "locations": 1}}

Each record message contains a stream identifier which specifies the unique name of that data. For data coming from an API this can be thought of as the name of the endpoint. For data coming from a database this might be the table name. The schema message will have a matching stream identifier for the records it describes. The term "stream" will be used in the rest of the documentation to identify a set of data being extracted.

Taps#

These 3 message types are generated by programs called taps. A tap can be written in any programming language (note that Meltano will only run Python-based taps). Taps output the 3 messages to standard output, aka stdout.

Taps are required to accept 1 file, called a configuration (config) file, and can optionally accept 2 other files called state and catalog files.

Targets#

The 3 message types are consumed by programs called targets. A target can be written in any programming language (note that Meltano will only run Python-based targets). Targets ingest the 3 messages from standard input, aka stdin.

Targets can optionally accept a configuration file if the target system requires authentication information. For a simple target like a csv file, this would not be required, but for a more complicated target like a SaaS database, the config file would be required.

Taps | Targets#

Since taps and targets are able to communicate to each other via the Singer spec, they can be used together to move data between systems. This can be done on the command line by sending the messages from a tap to a target using a Unix pipe, |. A pipe takes information from stdout of one process, in this case a tap, and redirects it to stdin of a second process, in this case a target. This composability means taps and targets can be composed as simply as tap | target.

Details#

Messages#

Each of the messages have a defined schema and some required and optional fields. Note that while example messages will be shown on multiple lines, each record when output from a tap must be on its own line.

Records#

Record messages contain the actual data being extracted from a source system. Every record message must have the following properties:

• type - this will always be RECORD
• stream - the unique identifier of the data stream
• record - a JSON object containing the data being extracted

Record messages can optionally have:

• time_extracted - The time the record was observed in the source. This should be an RFC3339 formatted date-time, like "2022-11-20T16:45:33.000Z".

Putting it together, a full record message looks like this:

{
  "type": "RECORD",
  "stream": "tools",
  "time_extracted": "2021-11-20T16:45:33.000Z",
  "record": {
    "id": 1,
    "name": "Meltano",
    "active": true,
    "updated_at": "2021-10-20T16:45:33.000Z"
  }
}

Note that in the above example the message was formatted for readibility, but when output from a tap the entire message will be on a single-line.

Schemas#

Schema messages define the structure of the data sent in a record message. Every schema message must have the following properties:

• type - this will always be SCHEMA
• stream - the unique identifier of the data stream. This will match the stream property in record messages
• schema - a JSON Schema describing the record property of record messages for a given stream
• key_properties - a list of strings indicating which properties make up the primary key for this stream. Each item in the list must be the name of a top-level property defined in the schema. An empty list may be used to indicate there is no primary key for the stream

What is a JSON Schema?#

A JSON Schema is a way to annotate and validate JSON objects. The data types available in raw JSON are limited compared to the variety of types available in many targets. Within the Singer Spec, JSON schema definitions are used to tell a target the exact data type to use when storing data.

Using the record example shown previously, the JSON schema for that record could be:

{
    "properties": {
      "id": {
        "type": "integer"
      },
      "name": {
        "type": "string"
      },
      "active": {
        "type": "boolean"
      },
      "updated_at": {
        "type": "string",
        "format": "date-time"
      }
    }

This definition now explicitly defines what kind of data is expected in a record and how to handle it when loading the data.

Also of note, there are several different versions of JSON Schema. The most common one is Draft 4 and Meltano and the SDK both support this draft.

Optional SCHEMA message properties#

Schema messages can optionally have:

• bookmark_properties - a list of strings indicating which properties the tap is using as bookmarks. Each item in the list must be the name of a top-level property defined in the schema. This is discussed more in the bookmarks section.

Putting it together, a full schema message looks like this:

{
  "type": "SCHEMA",
  "stream": "tools",
  "schema": {
    "properties": {
      "id": {
        "type": "integer"
      },
      "name": {
        "type": "string"
      },
      "active": {
        "type": "boolean"
      },
      "updated_at": {
        "type": "string",
        "format": "date-time"
      }
    }
  },
  "key_properties": ["id"],
  "bookmark_properties": ["updated_at"]
}

Note that in the above example the message was formatted for readability, but when output from a tap the entire message will be on a single-line.

Ordering of SCHEMA and RECORD Messages#

Before any record messages for a given data stream are output by a tap, they must be preceded by a schema message for the stream. While the extraction will still work, it will be assumed that the record is schema-less and will be loaded in a potentially unexpected manner.

State#

State messages contain any information that a tap is designed to persist. These are used to inform the target of the current place in the extraction of a data stream. Each state message must have the following properties:

• type - this will always be STATE
• value - this is a JSON object of the state values to be stored

There structure of the value property is not defined by the spec and is determined by each tap independently. However, the following structure is recommended:

{
  "bookmarks": {
    "tools": {
      "updated_at": "2021-10-20T16:45:33.000Z"
    },
    "team": {
      "id": 123
    }
  }
}

The bookmarks key should be familiar since it's an optional key in a schema message. Each property within the bookmarks JSON object is a data stream from a previously defined schema and record. Each stream maps to a JSON object storing the last data point seen in the extraction.

In the above example, the tools stream has extracted data up to the timestamp shown in the updated_at field. Similarly, the team stream has extracted up to id = 123.

Putting it together, a full state message looks like this:

{
  "type": "STATE",
  "value": {
    "bookmarks": {
      "tools": {
        "updated_at": "2021-10-20T16:45:33.000Z"
      },
      "team": {
        "id": 123
      }
    }
  }
}

Note that in the above example the message was formatted for readibility, but when output from a tap the entire message will be on a single-line.

Taps#

When taps are run, they can accept three files that provide information necessary for it to work properly: config, state, and catalog files. Taps are required to accept the config file, and can optionally accept the state and catalog files.

Config Files#

The config file contains the parameters required by the tap to succesfully extract data from the source. Typically this will include credentials for an API or database connection.

There is no required specification, but it is recommended to have the following fields:

• start_date - this is used on the first sync to define how far back in time to pull data. Start dates should conform to the RFC3339 specification.
• user_agent - this should be an email address or other contact information should the API provider need to contact you for any reason

Putting this all together, a config file may look like:

# config.json
{
  "api_key" : "asd23ayzxz80adf",
  "start_date" : "2022-01-01T00:00:00Z",
  "user_agent" : "your_email@domain"
}

State Files#

Taps can optionally use a state file to start replication from a previous point in a data stream. The structure of the state file and the state message described previously should be nearly identical. The value property in the state message will be the contents of any state.json file passed to a tap.

Using the previous example, a state file would look like this:

# state.json
{
  "bookmarks": {
    "tools": {
      "updated_at": "2021-10-20T16:45:33.000Z"
    },
    "team": {
      "id": 123
    }
  }
}

Catalog Files#

Catalog files define the structure of one or many data streams. Taps are capable of both using and generating catalog files.

The structure of a catalog file is a JSON object with a single top-level property:

• streams - this is a list containing information for each data stream that can be extracted

Each item within the streams list is another JSON object with the following required properties:

• stream - this is the primary identifier of the stream as it will be passed to the target (tools, team, etc.)
• tap_stream_id - this is the unique identifier of the stream which can differ from the stream name since some sources may have multiple available streams with the same name
• schema - this is the JSON schema of the stream, which will be passed in a SCHEMA message to the Target

Optional properties within the list are:

• table_name - this is only used for a database source and is the name of the table
• metadata - this is a list that defines extra information about items within a stream. This is discussed more in the Metadata section below

An example catalog with a single stream and no metadata is as follows:

{
  "streams": [
    {
      "stream": "tools",
      "tap_stream_id": "tools",
      "schema": {
        "type": ["null", "object"],
        "additionalProperties": false,
        "properties": {
          "id": {
            "type": [
              "string"
            ]
          },
          "name": {
            "type": [
              "string"
            ]
          },
          "updated_at": {
            "type": [
              "string"
            ],
            "format": "date-time"
          }
        }
      }
    }
  ]
}

Discovery Mode#

Discovery mode is how taps can generate catalogs. When a tap is invoked with a --discover it will output the full catalog list of streams available for extraction to stdout. This can then be saved to a catalog.json file.

tap --config config.json --discover > catalog.json

Note that some older taps use properties.json as the catalog file.

Metadata#

Metadata is the preferred method of associating extra information about streams and properties within a stream.

There are two kinds of metadata:

• discoverable - this metadata should be written and read by a tap

• non-discoverable - this metadata is written by other systems, such as Meltano, and should only be read by the tap

  A tap is free to write any type of metadata they feel is useful for describing fields in the schema, although several reserved keywords exist. A tap that extracts data from a database should use additional metadata to describe the properties of the database.

Non-discoverable Metadata

Discoverable Metadata

Each piece of metadata has two primary keys:

• metadata - this is a JSON object containing all of the metadata for either the stream or a property of the stream
• breadcrumb - this identifies whether the metadata applies to the entire stream or a property of the stream. An empty list means the metadata applies to the stream. For specific properties within the stream, the breadcrumb will have the properties key followed by the name of the property being described.

An example of a valid metadata object is as follows:

"metadata": [
  {
    "metadata": {
      "inclusion": "available",
      "table-key-properties": ["id"],
      "selected": true,
      "valid-replication-keys": ["date_modified"],
      "schema-name": "users",
    },
    "breadcrumb": []
  },
  {
    "metadata": {
      "inclusion": "automatic"
    },
    "breadcrumb": ["properties", "id"]
  },
  {
    "metadata": {
      "inclusion": "available",
    "selected": true
    },
    "breadcrumb": ["properties", "name"]
  },
  {
    "metadata": {
      "inclusion": "automatic"
    },
    "breadcrumb": ["properties", "updated_at"]
  }
]

Putting it Together#

Putting this all together, a complete catalog example looks like this:

{
  "streams": [
    {
      "stream": "tools",
      "tap_stream_id": "tools",
      "schema": {
        "type": ["null", "object"],
        "additionalProperties": false,
        "properties": {
          "id": {
            "type": [
              "string"
            ],
          },
          "name": {
            "type": [
              "string"
            ],
          },
          "updated_at": {
            "type": [
              "string"
            ],
            "format": "date-time",
          }
        }
      }
    }
  ],
  "metadata": [
    {
      "metadata": {
        "inclusion": "available",
        "table-key-properties": ["id"],
        "selected": true,
        "valid-replication-keys": ["date_modified"],
        "schema-name": "users",
      },
      "breadcrumb": []
    },
    {
      "metadata": {
        "inclusion": "automatic"
      },
      "breadcrumb": ["properties", "id"]
    },
    {
      "metadata": {
        "inclusion": "available",
        "selected": true
      },
      "breadcrumb": ["properties", "name"]
    },
    {
      "metadata": {
        "inclusion": "automatic"
      },
      "breadcrumb": ["properties", "updated_at"]
    }
  ]
}

Metrics#

A tap can periodically emit structured log messages containing metrics about read operations. Consumers of the tap logs can parse these metrics out of the logs for monitoring or analysis. Metrics appear in the log output with the following structure:

INFO METRIC: <metrics-json>

where <metrics-json> is a JSON object with the following keys:

Here are some examples of metrics and how those metrics should be interpreted.

Timer for Successful HTTP GET#

INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 1.23, "tags": {"endpoint": "orders", "http_status_code": 200, "status": "succeeded"}}

The following is what the object looks like expanded:

{
    "type": "timer",
    "metric": "http_request_duration",
    "value": 1.23,
    "tags": {
        "endpoint": "orders",
        "http_status_code": 200,
        "status": "succeeded"
    }
}

This can be interpreted as: an HTTP request to an "orders" endpoint was made that took 1.23 seconds and succeeded with a status code of 200.

Timer for Failed HTTP GET#

INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 30.01, "tags": {"endpoint": "orders", "http_status_code": 500, "status": "failed"}}

This can be interpreted as: an HTTP request to an "orders" endpoint was made that took 30.01 seconds and failed with a status code of 500.

Counter for Records#

INFO METRIC: {"type": "counter", "metric": "record_count", "value": 100, "tags": {"endpoint": "orders"}}
INFO METRIC: {"type": "counter", "metric": "record_count", "value": 100, "tags": {"endpoint": "orders"}}
INFO METRIC: {"type": "counter", "metric": "record_count", "value": 100, "tags": {"endpoint": "orders"}}
INFO METRIC: {"type": "counter", "metric": "record_count", "value": 14, "tags": {"endpoint": "orders"}}

This can be interpreted as: a total of 314 records were fetched from an "orders" endpoint.

Log Output#

Metrics messages are interspersed with the primary 3 messages, so parsing them should be handled programmatically. This is an example of what a realistic log output might look like:

INFO Using API Token authentication.
INFO tickets: Skipping - not selected
{"type": "SCHEMA", "stream": "groups", "schema": {"properties": {"name": {"type": ["string"]}, "created_at": {"format": "date-time", "type": ["string"]}, "url": {"type": ["string"]}, "updated_at": {"format": "date-time", "type": ["string"]}, "deleted": {"type": ["boolean"]}, "id": {"type": ["integer"]}}, "type": ["object"]}, "key_properties": ["id"]}
INFO groups: Starting sync
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.6276309490203857, "tags": {"status": "succeeded"}}
{"type": "RECORD", "stream": "groups", "record": {"id": 360007960773, "updated_at": "2020-01-09T09:57:16.000000Z"}}
{"type": "STATE", "value": {"bookmarks": {"groups": {"updated_at": "2020-01-09T09:57:16Z"}}}}

Targets#

When targets are run, they can accept a single config file that provides the information necessary for it to work properly.

Config Files#

Similar to taps, targets take a configuration file. There is no specification for the structure of a config file, as long as it is JSON based.

State Files#

Unlike taps, targets do not take a state file. Targets are expected to read in the state messages from stdin, but typically they do not do anything with the state messages beyond sending them to stdout. This is done once all data that appeared in the stream before the state message has been processed by the Target.

Schema Files#

Targets do not take a schema file. However, they are expected to read the schema messages from stdin and perform validation of the incoming record using the provided schema.