Continuous integration jobs in dbt Cloud

You can set up continuous integration (CI) jobs to run when someone opens a new pull request (PR) in your dbt Git repository. By running and testing only modified models, dbt Cloud ensures these jobs are as efficient and resource conscientious as possible on your data platform.

Set up CI jobs

dbt Labs recommends that you create your CI job in a dedicated dbt Cloud deployment environment that's connected to a staging database. Having a separate environment dedicated for CI will provide better isolation between your temporary CI schema builds and your production data builds. Additionally, sometimes teams need their CI jobs to be triggered when a PR is made to a branch other than main. If your team maintains a staging branch as part of your release process, having a separate environment will allow you to set a custom branch and, accordingly, the CI job in that dedicated environment will be triggered only when PRs are made to the specified custom branch. To learn more, refer to Get started with CI tests.

Prerequisites

• You have a dbt Cloud account.
• For the Concurrent CI checks and Smart cancellation of stale builds features, your dbt Cloud account must be on the Team or Enterprise plan.
• You must be connected using dbt Cloud’s native Git integration with GitHub, GitLab, or Azure DevOps.
  • With GitLab, you need a paid or self-hosted account which includes support for GitLab webhooks and project access tokens. With GitLab Free, merge requests will invoke CI jobs but CI status updates (success or failure of the job) will not be reported back to GitLab.
  • If you previously configured your dbt project by providing a generic git URL that clones using SSH, you must reconfigure the project to connect through dbt Cloud's native integration.

To make CI job creation easier, many options on the CI job page are set to default values that dbt Labs recommends that you use. If you don't want to use the defaults, you can change them.

1. On your deployment environment page, click Create Job > Continuous Integration Job to create a new CI job.

2. Options in the Job Description section:

   • Job Name — Specify the name for this CI job.
   • Environment — By default, it’s set to the environment you created the CI job from.
   • Triggered by pull requests — By default, it’s enabled. Every time a developer opens up a pull request or pushes a commit to an existing pull request, this job will get triggered to run.
     • Run on Draft Pull Request — Enable this option if you want to also trigger the job to run every time a developer opens up a draft pull request or pushes a commit to that draft pull request.

3. Options in the Execution Settings section:

   • Commands — By default, it includes the dbt build --select state:modified+ command. This informs dbt Cloud to build only new or changed models and their downstream dependents. Importantly, state comparison can only happen when there is a deferred environment selected to compare state to. Click Add command to add more commands that you want to be invoked when this job runs.

   • Compare changes against an environment (Deferral) — By default, it’s set to the Production environment if you created one. This option allows dbt Cloud to check the state of the code in the PR against the code running in the deferred environment, so as to only check the modified code, instead of building the full table or the entire DAG.

     info

     Older versions of dbt Cloud only allow you to defer to a specific job instead of an environment. Deferral to a job compares state against the project code that was run in the deferred job's last successful run. While deferral to an environment is more efficient as dbt Cloud will compare against the project representation (which is stored in the manifest.json) of the last successful deploy job run that executed in the deferred environment. By considering all deploy jobs that run in the deferred environment, dbt Cloud will get a more accurate, latest project representation state.

   • Generate docs on run — Enable this option if you want to generate project docs when this job runs. This option is disabled by default since most teams do not want to test doc generation on every CI check.

   Example of CI Job page in dbt Cloud UI

4. (optional) Options in the Advanced Settings section:

   • Environment Variables — Define environment variables to customize the behavior of your project when this CI job runs. You can specify that a CI job is running in a Staging or CI environment by setting an environment variable and modifying your project code to behave differently, depending on the context. It's common for teams to process only a subset of data for CI runs, using environment variables to branch logic in their dbt project code.
   • Target Name — Define the target name. Similar to Environment Variables, this option lets you customize the behavior of the project. You can use this option to specify that a CI job is running in a Staging or CI environment by setting the target name and modifying your project code to behave differently, depending on the context.
   • Run Timeout — Cancel this CI job if the run time exceeds the timeout value. You can use this option to help ensure that a CI check doesn't consume too much of your warehouse resources.
   • dbt Version — By default, it’s set to inherit the dbt version from the environment. dbt Labs strongly recommends that you don't change the default setting. This option to change the version at the job level is useful only when you upgrade a project to the next dbt version; otherwise, mismatched versions between the environment and job can lead to confusing behavior.
   • Threads — By default, it’s set to 4 threads. Increase the thread count to increase model execution concurrency.
   • Run source freshness — Enable this option to invoke the dbt source freshness command before running this CI job. Refer to Source freshness for more details.
   Example of Advanced Settings on the CI Job page

Trigger a CI job with the API

If you're not using dbt Cloud’s native Git integration with GitHub, GitLab, or Azure DevOps, you can use the Administrative API to trigger a CI job to run. However, dbt Cloud will not automatically delete the temporary schema for you. This is because automatic deletion relies on incoming webhooks from Git providers, which is only available through the native integrations.

Prerequisites

• You have a dbt Cloud account.
• For the Concurrent CI checks and Smart cancellation of stale builds features, your dbt Cloud account must be on the Team or Enterprise plan.

1. Set up a CI job with the Create Job API endpoint using "job_type": ci or from the dbt Cloud UI.

2. Call the Trigger Job Run API endpoint to trigger the CI job. You must include these fields to the payload:

   • Provide the pull request (PR) ID with one of these fields, even if you're using a different Git provider (like Bitbucket). This can make your code less human-readable but it will not affect dbt functionality.

     • github_pull_request_id
     • gitlab_merge_request_id
     • azure_devops_pull_request_id 

   • Provide the git_sha or git_branch to target the correct commit or branch to run the job against.

Example pull requests

The green checkmark means the dbt build and tests were successful. Clicking on the dbt Cloud section navigates you to the relevant CI run in dbt Cloud.

GitHub pull request example

GitHub pull request example

GitLab pull request example

GitLab pull request

Azure DevOps pull request example

Azure DevOps pull request

Troubleshooting

If you're experiencing any issues, review some of the common questions and answers below.

Temporary schemas aren't dropping

If your temporary schemas aren't dropping after a PR merges or closes, this typically indicates one of these issues:

• You have overridden the generate_schema_name macro and it isn't using dbt_cloud_pr_ as the prefix.
  
  To resolve this, change your macro so that the temporary PR schema name contains the required prefix. For example:
  
  ✅ Temporary PR schema name contains the prefix dbt_cloud_pr_ (like dbt_cloud_pr_123_456_marketing).
  ❌ Temporary PR schema name doesn't contain the prefix dbt_cloud_pr_ (like marketing).
  


• A macro is creating a schema but there are no dbt models writing to that schema. dbt Cloud doesn't drop temporary schemas that weren't written to as a result of running a dbt model.

Reconnecting your dbt project to use dbt Cloud's native integration with GitHub, GitLab, or Azure DevOps

If your dbt project relies the generic git clone method that clones using SSH and deploy keys to connect to your dbt repo, you need to disconnect your repo and reconnect it using the native GitHub, GitLab, or Azure DevOps integration in order to enable dbt Cloud CI.

First, make sure you have the native GitHub authentication, native GitLab authentication, or native Azure DevOps authentication set up depending on which git provider you use. After you have gone through those steps, go to Account Settings, select Projects and click on the project you'd like to reconnect through native GitHub, GitLab, or Azure DevOps auth. Then click on the repository link.

Once you're in the repository page, select Edit and then Disconnect Repository at the bottom.

Confirm that you'd like to disconnect your repository. You should then see a new Configure a repository link in your old repository's place. Click through to the configuration page:

Select the GitHub, GitLab, or AzureDevOps tab and reselect your repository. That should complete the setup of the project and enable you to set up a dbt Cloud CI job.

Error messages that refer to schemas from previous PRs

If you receive a schema-related error message referencing a previous PR, this is usually an indicator that you are not using a production job for your deferral and are instead using self. If the prior PR has already been merged, the prior PR's schema may have been dropped by the time the CI job for the current PR is kicked off.

To fix this issue, select a production job run to defer to instead of self.

Production job runs failing at the Clone Git Repository step

dbt Cloud can only check out commits that belong to the original repository. dbt Cloud cannot checkout commits that belong to a fork of that repository.

If you receive the following error message at the Clone Git Repository step of your job run:
Error message:
Cloning into '/tmp/jobs/123456/target'...
Successfully cloned repository.
Checking out to e845be54e6dc72342d5a8f814c8b3316ee220312...
Failed to checkout to specified revision.
git checkout e845be54e6dc72342d5a8f814c8b3316ee220312
fatal: reference is not a tree: e845be54e6dc72342d5a8f814c8b3316ee220312


Double-check that your PR isn't trying to merge using a commit that belongs to a fork of the repository attached to your dbt project.

CI job not triggering for Virtual Private dbt users

To trigger jobs on dbt Cloud using the API, your Git provider needs to connect to your dbt Cloud account.

If you're on a Virtual Private dbt Enterprise plan using security features like ingress PrivateLink or IP Allowlisting, registering CI hooks may not be available and can cause the job to fail silently.