Jobs API

There're some operations that can start a job in Tinybird:

  • Import data via URL.
  • Populate a Data Source from a materialized node.
  • Run background queries.

When any of these operations start, the response contains a job_id field.

With the Jobs API, you can list the jobs for the last 48 hours or the last 100 jobs.

You can also get the details for a job. Depending on the job kind (import, query, or populate), it will return certain information related with the specific job, along with the status of the job.

The Job status can be one of the following:

  • waiting: The initial status of a job. When creating a job, it has to wait if there're other jobs running.
  • working: Once the job operation has started.
  • done: The job has finished successfully.
  • error: The job has finished with an error.

The created_at field reflects when the job was created, while updated_at shows when the job changed its state. Once the status of the job is done or error the difference between created_at and updated_at is the total execution time, including the time spent on the job queue.

The statistics fields shows: - row_count: For import and populate jobs, it shows the number of rows of the Data Source. In case of a query job, it shows the number of rows returned by the query. - bytes: For import and populate jobs, it shows the amount of bytes of the target Data Source. In case of a query job, the size, in bytes, returned by the query.

The quarantine_rows field shows the number of rows in the quarantine datasource. Those that do not match the schema but can still be parsed, while the invalid_lines field shows the number of rows that cannot be properly parsed.

GET /v0/jobs/?

We can get a list of the last 100 jobs in the last 48 hours:

We can also filter the jobs using the following parameters:

Request parameters

Key

Type

Description

kind

String

This will return only the jobs with that particular kind. Example: kind=populateview or kind=copy or kind=import

status

String

This will return only the jobs with the status provided. Example: status=done or status=waiting or status=working or status=error

pipe_id

String

This will return only the jobs associated with the provided pipe id. Example: pipe_id=t_31a0ff508c9843b59c32f7f81a156968

pipe_name

String

This will return only the jobs associated with the provided pipe name. Example: pipe_name=test_pipe

created_after

String

This will return jobs that were created after the provided date in the ISO 8601 standard date format. Example: created_after=2023-06-15T18:13:25.855Z

created_before

String

This will return jobs that were created before the provided date in the ISO 8601 standard date format. Example: created_before=2023-06-19T18:13:25.855Z

Getting the latest jobs
curl \
-H "Authorization: Bearer <token>" \
"https://api.tinybird.co/v0/jobs" \

You will get a list of jobs with the kind, status, id, and the url to access the specific information about that job.

Jobs list
{
    "jobs": [
        {
            "id": "c8ae13ef-e739-40b6-8bd5-b1e07c8671c2",
            "kind": "import",
            "status": "done",
            "created_at": "2020-12-04 15:08:33.214377",
            "updated_at": "2020-12-04 15:08:33.396286",
            "job_url": "https://api.tinybird.co/v0/jobs/c8ae13ef-e739-40b6-8bd5-b1e07c8671c2",
            "datasource": {
                "id": "t_31a0ff508c9843b59c32f7f81a156968",
                "name": "my_datasource_1"
            }
        },
        {
            "id": "1f6a5a3d-cfcb-4244-ba0b-0bfa1d1752fb",
            "kind": "import",
            "status": "error",
            "created_at": "2020-12-04 15:08:09.051310",
            "updated_at": "2020-12-04 15:08:09.263055",
            "job_url": "https://api.tinybird.co/v0/jobs/1f6a5a3d-cfcb-4244-ba0b-0bfa1d1752fb",
            "datasource": {
                "id": "t_49806938714f4b72a225599cdee6d3ab",
                "name": "my_datasource_2"
            }
        }
    ]
}

Job details in job_url will be available for 48h after its creation.

POST /v0/jobs/(.+)/cancel

With this endpoint you can try to cancel an existing Job. All jobs can be cancelled if they are in the “waiting” status, but you can’t cancel a Job in “done” or “error” status.

In the case of the job of type “populate”, you can cancel it in the “working” state.

After successfully starting the cancellation process, you will see two different status in the job: - “cancelling”: The Job can’t be immediately cancelled as it’s doing some work, but the cancellation will eventually happen. - “cancelled”: The Job has been completely cancelled.

A Job cancellation doesn’t guarantee a complete rollback of the changes being made by it, sometimes you will need to delete new inserted rows or datasources created.

The fastest way to know if a job is cancellable, is just reading the “is_cancellable” key inside the job JSON description.

Depending on the Job and its status, when you try to cancel it you may get different responses: - HTTP Code 200: The Job has successfully started the cancellation process. Remember that if the Job has now a “cancelling” status, it may need some time to completely cancel itself. This request will return the status of the job. - HTTP Code 404: Job not found. - HTTP Code 403: The token provided doesn’t have access to this Job. - HTTP Code 400: Job is not in a cancellable status or you are trying to cancel a job which is already in the “cancelling” state.

Try to cancel a Job
curl \
-H "Authorization: Bearer <token>" \
-X POST "https://api.tinybird.co/v0/jobs/:job_id/cancel"
Populate Job in cancelling state right after the cancellation request.
{
    "kind": "populateview",
    "id": "32c3438d-582e-4a6f-9b57-7d7a3bfbeb8c",
    "job_id": "32c3438d-582e-4a6f-9b57-7d7a3bfbeb8c",
    "status": "cancelling",
    "created_at": "2021-03-17 18:56:23.939380",
    "updated_at": "2021-03-17 18:56:44.343245",
    "is_cancellable": false,
    "datasource": {
        "id": "t_02043945875b4070ae975f3812444b76",
        "name": "your_datasource_name",
        "cluster": null,
        "tags": {},
        "created_at": "2020-07-15 10:55:12.427269",
        "updated_at": "2020-07-15 10:55:12.427270",
        "statistics": null,
        "replicated": false,
        "version": 0,
        "project": null,
        "used_by": []
    },
    "query_id": "01HSZ9WJES5QEZZM4TGDD3YFZ2",
    "pipe_id": "t_7fa8009023a245b696b4f2f7195b23c3",
    "pipe_name": "top_product_per_day",
    "queries": [
        {
            "query_id": "01HSZ9WJES5QEZZM4TGDD3YFZ2",
            "status": "done"
        },
        {
            "query_id": "01HSZ9WY6QS6XAMBHZMSNB1G75",
            "status": "done"
        },
        {
            "query_id": "01HSZ9X8YVEQ0PXA6T2HZQFFPX",
            "status": "working"
        },
        {
            "query_id": "01HSZQ5YX0517X81JBF9G9HB2P",
            "status": "waiting"
        },
        {
            "query_id": "01HSZQ6PZJA3P81RC6Q6EF6HMK",
            "status": "waiting"
        },
        {
            "query_id": "01HSZQ76D7YYFB16TFT32KXMCY",
            "status": "waiting"
        }
    ],
    "progress_percentage": 50.0
}