How to Run Background Tasks Using the User Interface (UI)

The following page explains how to create and run a Background Task using the UI. In order to create a Background Task, a user must have the TASK_EDITOR permission. In addition, a TASK_EDITOR can update and delete any task when the user also has assigned ownership. For more information, see the Background Tasks page or the Permissions and Roles page.

Table of contents

• How to Create a Background Task
  • Create a Background Task: Query to Source
  • Create a Custom Background Task
• Run a Background Task
  • View Task Run Results
• Update a Background Task
• Delete a Background Task

How to Create a Background Task

When creating a new Background Task, users can choose between “Query To Source” or “Custom.” The “Query To Source” Background Task pattern allows users to automate data analysis tasks with minimal inputs, reducing the need for development resources, and without needing to call the API. The “Custom” Background Task option is the existing functionality that permits users to create and execute tasks based on images uploaded to the Elastic Container Registry (ECR) repository.

Note. Running Background Tasks is subject to permissions checks. See Background Task Permissions for details.

Create a Background Task: Query to Source

The “Query To Source” pattern enables data engineers to automate data analysis and filtering tasks, eliminates the need to create container images for routine data operations, and increases operational efficiency and scalability. When a Query To Source Background Task runs, a query is executed using the configured SQL. The results of that query is then uploaded to the specified Source.

To create a Query to Source Background Task using the UI, a user must log in to Data Refinery (DR) Designer with their username and password. Follow the steps below to add a new Query to Source task.

1. Once logged in, select the Tasks tab in the navigational bar at the top of the page.

   

2. When the page has loaded, click the Create Task button to the right of the page.

   

3. A dropdown will appear. Select Query to Source.

   

4. Next, a form will appear. Fill in all of the fields.

   

   • Name: Preferred Name of the Background Task.
     • May only contain alphanumeric characters, dashes, or underscores.
     • Must be no more than 50 characters.
   • Query: SQL Query to run.
   • Source: Source in which to upload the resulting dataset.
   • Create New Source Version: Indicates whether to create a new version for the Source.
   • Schedule: Written in the form of a Cron string. It must follow the standard crontab specs. Background Tasks are scheduled with Coordinated Universal Time (UTC).

5. Once the fields in the form are completed, click Create.

Create a Custom Background Task

To create a Custom Background Task using the UI, a user must log in to DR Designer with their username and password. Follow the steps below to add a new custom task.

1. Once logged in, select the Tasks tab in the navigational bar at the top of the page.

   

2. When the page has loaded, click the Create Task button to the right of the page.

   

3. A dropdown will appear. Select Custom.

   

4. Next, a form will appear. Fill in all of the fields.

   

   • Name: Preferred Name of the Background Task.
     • May only contain alphanumeric characters, dashes, or underscores.
     • Must be no more than 50 characters.
   • Image: Use the repository URL and image tag to build the Background Task image.
     • Must be no more than 255 characters.
   • Schedule: Written in the form of a Cron string. It must follow the standard crontab specs. Background Tasks are scheduled with Coordinated Universal Time (UTC).

   Note. To build and upload a Background Task image, refer to the Uploading the Background Task Image section.

5. Once the fields in the form are completed, click Create.

Note. Some Background Task attributes, such as environment variables, are not viewable and editable in the Create Task form.

Run a Background Task

A Background Task normally runs on its optional cron schedule, using Coordinated Universal Time (UTC). But whether or not a schedule exists, a user can manually execute a Background Task. To manually execute a Background Task after creating it, a user can select the Run button in the Action column of the Tasks list.

After selecting the Run task button, DR Designer will notify the user that the execution for the Background Task was successful. See image below.

To run your task with the API, refer to the How to Run Background Tasks Using APIs page.

View Task Run Results

A user can access a Background Task’s executions list to view the 50 most recent executions in Data Refinery.

These executions can be viewed regardless of task ownership with the following steps.

1. Once logged in, select the Tasks tab in the navigational bar at the top of the page.

   

2. Find the desired task in the Tasks list or by name in the search bar.

3. In the Action column to the right of the listed task, select the View Executions button.

   

   This action will open a new tab with a list of the 50 most recent executions for that Background Task. This list contains the following details:

   • Execution ID: ID of the execution.
   • Definition Type: Either AWS or Local.
   • Task Identifiers: ID of the container the task is running in.
   • Status: Will show “Failed,” “Running,” or “Successful” status.
   • Finished At: A timestamp of when the task execution completed.
   • Exit Code: This number represents the status of the task after completion.
     • Zero: Successful
     • Non-Zero: Failed
     • Empty: Still Running

   From this page, a user can also run the Background Task and refresh the executions list.

To view task run results via the API, refer to the View Task Run Results section.

Update a Background Task

To update a Background Task in the UI, a user must be signed in to DR Designer. Once a user has navigated to the Tasks tab, follow the procedure below.

1. Find the desired task in the Tasks list.

2. In the Action column to the right of the listed task, select the Update button.

   

3. A form will appear. Edit the desired fields and select the Cancel or Update button to continue.
4. Any changes made to the task will be updated automatically.

Note. Some Background Task attributes, such as environment variables, are not viewable and editable in the Update Task form.

Delete a Background Task

To delete a Background Task in the UI, a user must be signed in to DR Designer and hold task ownership. Once a user has navigated to the Tasks tab, follow the procedure below.

1. Find the desired task in the Tasks list.

2. In the Action column to the right of the listed task, select the Delete button.

   

3. A dialog box will appear to warn the user that the task will be deleted and the action cannot be undone. Select the Delete button to confirm the deletion or the Cancel button to return to the Tasks list.

   

   Upon selection, the user will return to the Tasks list.