How to Run Background Tasks Using APIs

To run a Background Task using the API endpoints, a user interacts with the refinery through HTTP requests. The refinery provides specific endpoints to create and run Background Tasks. Below are the steps to run a Background Task using the APIs.

In order to create a Background Task, a user must have the TASK_EDITOR permission. For more information, see the Background Tasks page or the Permissions and Roles page.

Table of contents

Selecting a Launch Template

The application ships with default launch templates that are pre-configured to support Background Task execution seamlessly. Users can take advantage of these pre-defined templates without specifying all the details explicitly. The required Virtual Private Cloud (VPC) information and target Elastic Container Service (ECS) cluster are pulled from the application’s configuration.

To select from an existing launch template, use the curl command below to send a HTTP GET request to /ecs/launchtemplates endpoint to retrieve the list of available templates. Once the list is rendered, choose the relevant template based on the Background Task requirements.

   curl -X GET "https://example.kingland-data-refinery.com/api/ecs/launchtemplates" \ 
   -H "Authorization: Bearer $JWT_TOKEN"

This API returns a list of available launch templates to choose from. An example return response is shown below.

   [
    {"ID":1, "templateName":"default-template", "cluster":"example-cluster-name"},
    {"ID":2, "templateName":"hpc-instances", "cluster":"example-cluster-name"},
   ]

Try retrieving the list of available templates using the DR Designer API Reference.

Uploading the Background Task Image

To upload a Background Task image to the Elastic Container Registry (ECR) repository, follow these steps to send HTTP requests to the refinery.

  1. If required, retrieve and set the API authorization token by following the steps in Appendix A: Retrieve and Set JWT_TOKEN.

  2. To fetch the ECR repository URL, send a HTTP GET request to /tasks/repo/url endpoint using the following curl command. 
    curl -X GET "https://example.kingland-data-refinery.com/api/tasks/repo/url" \ 
-H "Authorization: Bearer $JWT_TOKEN"

    This returns the ECR repository URL string as shown in the example below.

    example-account.dkr.ecr.example-region.amazonaws.com/bgtask_example

    Try fetching the ECR repository URL using the DR Designer API Reference.

  3. Next, fetch the ECR repository password by sending a HTTP GET request to /tasks/repo/token using the following curl command.

    curl -X GET "https://example.kingland-data-refinery.com/api/tasks/repo/token" \ 
-H "Authorization: Bearer $JWT_TOKEN"

    This returns the token required to configure the Amazon Web Services, Command Line Interface (AWS CLI) and the role to assume. See below.

    {"password": "example-ecr-repo-token", "expiresAt": "yyyy-mm-ddThh:mm:ss.SSSZ"}

    Try fetching the ECR repository password using the DR Designer API Reference.

  4. Docker should be installed on the user’s machine to log in to the ECR repository by using the repo URL and password retrieved in Step 2 and Step 3, respectively.

    docker login example-account.dkr.ecr.example-region.amazonaws.com --username AWS --password your-password

    Replace “your-password” with the password retrieved from Step 3. If successful, the “Login Succeeded” response will appear.

  5. Next, build the Docker image. Then, after the build is complete, tag your image to push to the repository.

    If an example Docker image is needed for this demo, refer to Appendix B: Using a Sample Docker Image.

    docker build -t bgtask_example .
docker tag bgtask_example:latest your-ecr-repo-url:latest-example

    Replace “your-ecr-repo-url” with the real repo URL retrieved in Step 2.

  6. Push this image to the ECR Background Task repository. 
    docker push your-ecr-repo-url:latest-example

    Replace “your-ecr-repo-url” with the real repo URL retrieved in Step 2.

Creating a New Task

To create a new Background Task using curl commands, follow these steps to send an HTTP POST request to the refinery.

  1. Prepare the task input in a JSON file based on the AWSBackgroundTask model. The JSON file should include the necessary fields for task definition including name, cronSchedule, imageUrl, envVars, cpu, memory, command, and reference the launch template found using Selecting a Launch Template.

       {
       "name": "example-task-name",
       "cronSchedule": "* * * * *",
       "imageUrl": "account.dck.ecr.region.amazonaws.com/bgtask_dev:latest-example",
       "cpu": 256,
       "memory": 512,
       "command": [""],
       "envVars": {
         "VAR1": "value1",
         "VAR2": "value2"
       },
       "launchTemplateModel": {
         "ID": 1,
         "templateName": "example-template-name",
         "cluster": "example-cluster-name"
       } 
   }

    Note. The example image from Appendix B: Using a Sample Docker Image, needs the following for command: ["/bin/sh", "-c", "echo 'hello from bgtask'"].

  2. If required, retrieve and set the API authorization token by following the steps in Appendix A: Retrieve and Set JWT_TOKEN.

  3. Use the curl command to send an HTTP POST request to the /tasks endpoint. Use the JSON file created in Step 1 as the task input. Include the necessary headers, such as the content type and authorization with the bearer token retrieved in Step 2.

    curl -X POST "https://example.kingland-data-refinery.com/api/tasks" \ 
-H "Content-Type: application/json" \
-H "Authorization: Bearer $JWT_TOKEN" \
-d "@PATH_TO_TASK_JSON_FILE"

    Below is an example of an expected task response.

     {
 "ID": 98,
 "name": "example-task-name",
 "cronSchedule": "* * * * *",
 "imageUrl": "1234567890.dkr.ecr.us-east-2.amazonaws.com/bgtask_dev:latest-example",
 "envVars": {
     "VAR1": "",
     "VAR2": ""
 },
 "taskDefinitionArn": "arn:aws:ecs:us-east-2:1234567890:task-definition/drf-bg:2",
 "launchType": 0,
 "cpu": 256,
 "memory": 512,
 "tags": null,
 "launchTemplate": 110,
 "launchTemplateModel": {
     "ID": 110,
     "templateName": "example-template-name",
     "cluster": "example-cluster-name"
 }
}

    Try creating a new task using the DR Designer API Reference.

Note. For security, Data Refinery Designer API redacts the values of a task’s environment variables’ values. Only the variable keys are returned. When updating the tasks, if EnvVars only contain empty values, then the EnvVars input is ignored and not updated.

Running the Task Manually

To run a Background Task manually, follow these steps to send a HTTP POST request to the refinery. Both scheduled and unscheduled tasks can be run manually.

  1. If required, retrieve and set the API authorization token by following the steps in Appendix A: Retrieve and Set JWT_TOKEN
  2. To run a Background Task, send a HTTP POST request to /tasks/:ID/execution endpoint using the following curl command. 
    curl -X POST "https://example.kingland-data-refinery.com/api/tasks/example-task-id/execution" \ 
-H "Authorization: Bearer $JWT_TOKEN"

    The run task response contains the taskIdentifier which can be used to check the task run results.

    {"ID":15,"definitionID":9,"definitionType":1,"taskIdentifier":"arn:aws:ecs:us-east-2:1234567890:task/example-cluster-name/example-task-arn"}

    Try running the task using the DR Designer API Reference.

View Task Run Results

To view the task run status, use the AWS CLI describe-tasks command to verify the desired status from the desired task. To ensure a successful run, check the exit code of the task container using the command below.

aws ecs describe-tasks --region your-aws-region\
  --cluster $STRING_CLUSTER_NAME \
  --tasks your-task-identifier \
  --query "tasks[0].containers[?name=='aws-bg-task'].exitCode" \
  --output text

Replace “your-aws-region” with AWS region and “your-task-identifier’ with the taskIdentifier value retrieved from Step 2 in Running the Task.

Troubleshoot Failed Scheduled Tasks

Scheduled tasks are started and tracked by the API containers. A new or updated task may take up to 30 seconds before the initial execution. The application’s CloudWatch logs contain events for when a task is scheduled, updated, or removed from the internal scheduler. If a task’s cron is removed, it becomes a single-execution task and is removed from the scheduler.

The logs’ format is [INFO] - {Registered|Removed} task from scheduler taskID={ID}.

Note. An update event is represented by a “Removed” event, then immediately followed by a “Registered” event.

Successfully started tasks are also logged in CloudWatch. The log’s format is: [INFO] - Background task started taskID={ID}.

Scheduled tasks may fail to start due to improper configuration, such as a bad cron schedule or resource allocation. If a scheduled task fails to start, it will not be queryable via the aws ecs describe-tasks command. Instead, an application log will be sent to CloudWatch detailing the failed task and more information on the failure. A CloudWatch alarm can be set up to trigger based on this error message if desired.

The log’s format is: [WARN] - Background task execution failed for taskID={ID} err={error message}.

Appendices

Appendix A: Retrieve and Set JWT_TOKEN

  1. To retrieve the bearer token, a user must log in using their username and password. Next, follow the steps in How to Authenticate.

  2. Replace YOUR_JWT_TOKEN_VALUE with the bearer token retrieved from login to set JWT_TOKEN.

    export JWT_TOKEN=YOUR_JWT_TOKEN_VALUE

Appendix B: Using a Sample Docker Image

To build a simple Docker image for a demo purpose, follow the steps below to pull and tag an alpine image.

docker pull alpine:latest
docker tag alpine:latest your-ecr-repo-url:latest-example

Replace “your-ecr-repo-url” fetch from Step 2 under Uploading the Background Task Image.

Appendix C: AWSBackgroundTask Model 

{
  "name": "string",
  "cronSchedule": "string",
  "imageUrl": "string",
  "envVars": {
    "key1": "value1",
    "key2": "value2",
  },
  "command": ["command1", "command2"],
  "taskDefinitionArn": "string",
  "launchType": 0,
  "cpu": 0,
  "memory": 0,
  "tags": [
    {
      "Key": "string",
      "Value": "string"
    },
    {
      "Key": "string",
      "Value": "string"
    },
  ],
  "launchTemplate": 0,
  "launchTemplateModel": {
    "templateName": "string",
    "cluster": "string"
  }
}

Copyright © 2025 Kingland Systems LLC