Sources
Full Example
contracts:
sources:
- filter:
- is_enabled
- name:
include: .*i\s+am\s+a\s+regex\s+pattern.*
exclude: .*i\s+am\s+a\s+regex\s+pattern.*
match_all: false
- meta:
meta:
key1: val1
key2:
- val2
- val3
validations:
- has_freshness
- has_allowed_meta_values:
meta:
key1: val1
key2:
- val2
- val3
- has_all_columns
- has_tests:
min_count: 1
max_count: 5
- has_required_tags:
tags: tag1
- has_description
- exists
- has_properties
- has_matching_description:
ignore_whitespace: true
case_insensitive: true
compare_start_only: false
generator:
exclude: columns
filename: properties.yml
depth: 1
description:
overwrite: false
terminator: \n
columns:
overwrite: true
add: true
remove: false
order: false
Filters
Filters (or Conditions) for reducing the scope of the contract. You may limit the number of sources processed by the rules of this contract by defining one or more of the following filters.
name
Filter sources based on their names.
Schema
name:
include:
default: []
description: Patterns to match against for values to include
items:
type: string
type: array
exclude:
default: []
description: Patterns to match against for values to exclude
items:
type: string
type: array
match_all:
default: false
description: When True, all given patterns must match to be considered a match
for either pattern type
type: boolean
Example
name:
include: .*i\s+am\s+a\s+regex\s+pattern.*
exclude: .*i\s+am\s+a\s+regex\s+pattern.*
match_all: false
You may also define the parameters for include directly on the definition like below.
name: .*i\s+am\s+a\s+regex\s+pattern.*
path
Filter sources based on their paths. Paths must match patterns which are relative to the root directory of the dbt project.
Schema
path:
include:
default: []
description: Patterns to match against for values to include
items:
type: string
type: array
exclude:
default: []
description: Patterns to match against for values to exclude
items:
type: string
type: array
match_all:
default: false
description: When True, all given patterns must match to be considered a match
for either pattern type
type: boolean
Example
path:
include: .*i\s+am\s+a\s+regex\s+pattern.*
exclude:
- ^\w+\d+\s{1,3}$
- exclude[_-]this
match_all: true
You may define the paths as a list of lists where each part is a subdirectory within the path. These parts will then be unified by joining them with the os-specific path separator. This allows for you define os-independent configuration as needed.
include:
- ["path", "to", "folder1"]
- ["path", "to", "folder2"]
- ["path", "to", "folder3"]
exclude:
- ["path", "to", "another", "folder1"]
- ["path", "to", "another", "folder2"]
- ["path", "to", "another", "folder3"]
You may also define the parameters for include directly on the definition like below.
path: .*i\s+am\s+a\s+regex\s+pattern.*
tag
Filter sources based on their tags.
Schema
tag:
tags:
default: []
description: The tags to match on
items:
type: string
type: array
Example
tag:
tags:
- tag1
- tag2
You may also define the parameters for tags directly on the definition like below.
tag: tag1
meta
Filter sources based on their meta values.
Schema
meta:
meta:
additionalProperties:
items:
type: string
type: array
description: The mapping of meta keys to their allowed values
type: object
Example
meta:
meta:
key1: val1
key2:
- val2
- val3
is_enabled
Filter sources taking only those which are enabled.
Note
This condition does not need further configuration. Simply define the condition’s name as an item in your configuration.
Validations
Validations (or Terms) to apply to the resources of this contract. These enforce certain standards that must be followed in order for the contract to be fulfilled.
has_properties
Check whether the sources have properties files defined.
Note
This term does not need further configuration. Simply define the term’s name as an item in your configuration.
has_description
Check whether the sources have descriptions defined in their properties.
Note
This term does not need further configuration. Simply define the term’s name as an item in your configuration.
has_required_meta_keys
Check whether the sources have the expected set of required meta keys set.
Schema
has_required_meta_keys:
keys:
default: []
description: The required meta keys
items:
type: string
type: array
Example
has_required_meta_keys:
keys:
- key1
- key2
You may also define the parameters for keys directly on the definition like below.
has_required_meta_keys: key1
has_allowed_meta_keys
Check whether the sources have only meta keys set from a configured permitted list.
Schema
has_allowed_meta_keys:
keys:
default: []
description: The allowed meta keys
items:
type: string
type: array
Example
has_allowed_meta_keys:
keys:
- key1
- key2
You may also define the parameters for keys directly on the definition like below.
has_allowed_meta_keys: key1
has_allowed_meta_values
Check whether the sources have only meta values set from a configured permitted mapping of keys to values.
Schema
has_allowed_meta_values:
meta:
additionalProperties:
items:
type: string
type: array
description: The mapping of meta keys to their allowed values
type: object
Example
has_allowed_meta_values:
meta:
key1: val1
key2:
- val2
- val3
exists
Check whether the sources exist in the database.
Note
This term does not need further configuration. Simply define the term’s name as an item in your configuration.
has_tests
Check whether sources have an appropriate number of tests configured.
Schema
has_tests:
min_count:
default: 1
description: The minimum count allowed.
minimum: 1
type: integer
max_count:
anyOf:
- exclusiveMinimum: 0
type: integer
- type: 'null'
default: null
description: The maximum count allowed.
Example
has_tests:
min_count: 2
max_count: 4
You may also define the parameters for min_count directly on the definition like below.
has_tests: 2
has_all_columns
Check whether sources have all columns set in their properties. Ensures that all columns present in the database are present in dbt project properties.
Note
This term does not need further configuration. Simply define the term’s name as an item in your configuration.
has_expected_columns
Check whether sources have the expected names of columns set in their properties. Also checks if those columns have the expected data types if configured to do so.
Schema
has_expected_columns:
columns:
anyOf:
- type: string
- items:
type: string
type: array
- additionalProperties:
type: string
type: object
default: []
description: A sequence of the names of the columns that should exist in the node,
or a mapping of the column names and their associated data types that should
exist.
Example
has_expected_columns:
columns:
- column1
- column2
- column3
has_matching_description
Check whether the descriptions configured in sources’ properties match the descriptions in the database.
Schema
has_matching_description:
ignore_whitespace:
default: false
description: Ignore any whitespaces when comparing data type keys.
type: boolean
case_insensitive:
default: false
description: Ignore cases and compare data type keys only case-insensitively.
type: boolean
compare_start_only:
default: false
description: Match data type keys when the two values start with the same value.
Ignore the rest of the data type definition in this case.
type: boolean
Example
has_matching_description:
ignore_whitespace: true
case_insensitive: true
compare_start_only: false
You may also define the parameters for ignore_whitespace directly on the definition like below.
has_matching_description: false
has_loader
Check whether sources have appropriate configuration for a loader in their properties.
Note
This term does not need further configuration. Simply define the term’s name as an item in your configuration.
has_freshness
Check whether sources have freshness configured in their properties.
Note
This term does not need further configuration. Simply define the term’s name as an item in your configuration.
has_downstream_dependencies
Check whether sources have an appropriate number of downstream dependencies.
Schema
has_downstream_dependencies:
min_count:
default: 1
description: The minimum count allowed.
minimum: 1
type: integer
max_count:
anyOf:
- exclusiveMinimum: 0
type: integer
- type: 'null'
default: null
description: The maximum count allowed.
Example
has_downstream_dependencies:
min_count: 2
max_count: 5
You may also define the parameters for min_count directly on the definition like below.
has_downstream_dependencies: 3
Generator
You may also configure a Generator for your contract. A Generator creates/updates properties files for sources from the attributes found on the database resource.
Schema
generator:
exclude:
default: []
description: The fields to exclude from the generated properties.
items:
enum:
- description
- columns
type: string
type: array
filename:
default: _config.yml
description: The name to give to new properties files generated by this generator.
type: string
depth:
anyOf:
- minimum: 0
type: integer
- type: 'null'
default: null
description: The depth at which to place newly generated files within the resource
folder. e.g. 0 would place the file in the root of the resource folder, 1 would
place it in a subfolder of the root of the resource folder, etc. By default,
the file is placed in the same folder in which the resource is stored.
description:
$ref: '#/$defs/SetNodeDescription'
default:
overwrite: true
terminator: null
description: Configuration for setting the description
columns:
$ref: '#/$defs/SetNodeColumns'
default:
overwrite: true
add: true
remove: true
order: true
description: Configuration for setting the columns
defs
SetNodeColumns:
overwrite:
default: true
description: Whether to overwrite existing properties with properties from the
database. When false, keeps the values already present in the properties if
present.
type: boolean
add:
default: true
description: Add columns to the properties file which are in the database object
but missing from the properties.
type: boolean
remove:
default: true
description: Remove columns from the properties file which are in the properties
file but not in the database object.
type: boolean
order:
default: true
description: Reorder columns in the properties file to match the order found in
the database object.
type: boolean
SetNodeDescription:
overwrite:
default: true
description: Whether to overwrite existing properties with properties from the
database. When false, keeps the values already present in the properties if
present.
type: boolean
terminator:
anyOf:
- type: string
- type: 'null'
default: null
description: Only take the description up to this terminating string. e.g. if
you only want to take the first line of a multi-line description, set this to
'\n'
Example
generator:
exclude: columns
filename: config.yml
depth: 1
description:
overwrite: true
terminator: __END__
columns:
overwrite: false
add: false
remove: true
order: true
You may also define the parameters for exclude directly on the definition like below.
generator:
- description
- columns
Columns
You may also define columns contracts as a child set of contracts on sources. Refer to the columns reference for more info.