Background Tasks
The Background Task execution feature provides users with a powerful tool to perform custom data operations in a flexible and efficient manner. This documentation details the process of creating, uploading, and executing Background Tasks, along with viewing the results. Additionally, this page will cover the steps to create and select a launch template for seamless execution.
Table of contents
Introduction to Background Tasks
Enabling Background Task features plays a crucial role in enhancing the refinery’s capabilities by offloading time-consuming and resource-intensive processes. Background Tasks are designed to be executed within a Docker container.
Users can choose to run the task on-demand as a “single-run” operation or schedule it for recurring executions using a cron schedule. Every Background Task receives its dedicated pool of hardware and software resources, ensuring seamless execution without any interference from other tasks.
For instance, users can leverage Background Tasks for data aggregation and synchronization, enabling them to run custom code that fetches data from various data repositories. Additionally, data processing and transformation tasks can be executed which facilitates the generation of Business Intelligence (BI) reports. This helps users build insightful dashboards. The Background Task feature empowers users to execute various data operations using their own code. It is a valuable tool for enhancing data analysis and decision-making processes.
This user documentation facilitates how to create a Background Task using the application APIs and also using the User Interface (UI). The following pages provide instructions for creating and selecting a launch template, uploading the Background Task image, creating a new task, running a task, and viewing the task results.
Background Task Permissions
In Data Refinery Designer (DR Designer), users who intend to work on Background Tasks must have the TASK_EDITOR permission. Currently, a user with no task permissions can only view the list of Background Tasks and run them. In comparison, users with the TASK_EDITOR permission will be able to create, update, and delete their Background Tasks in DR Designer.
Background Task Ownership
When a Background Task is created by a user with the TASK_EDITOR permission, they are also automatically assigned ownership of that task. Task ownership can only be held by one user and allows the user to update or delete the task.
To re-assign Background Task ownership, the recipient must have TASK_EDITOR permission and take task ownership, either by sending the PUT /tasks/:ID/owner API request or through the “Take Ownership” action button in the UI. Once task ownership is re-assigned, the original task creator is stripped of their ownership and can no longer update or delete the task.
For more information on visibility and access across DR Designer, see the Permissions and Roles page.
For more information on Task API calls, see the DR Designer API Tasks Reference.
Prebuilt Background Tasks for External Sources
Kingland continues to expand the capabilities of Data Refinery Designer by offering prebuilt Background Tasks. Analytics jobs commonly involve comparing client data to data from trusted, publicly available sources. To ensure that users of Data Refinery Designer can get analytics jobs up and running as quickly as possible, pre-built Background Tasks will retrieve data from key public sources and automatically upload that data into new a Source or new Source Version within Data Refinery.
Available Prebuilt Tasks
The following public data sources are available as prebuilt Background Tasks:
- Commodity Futures Trading Commission (CFTC)
- Financial Services Firms data
- European Union Central Bank (EUCB)
- Monetary Financial Institution data
- Federal Deposit Insurance Company (FDIC)
- Institutions data
- Locations data
- Investment Advisor Public Disclosure (IAPD)
- SEC Investment Advisors data
- State Investment Advisors data
- Investment Advisor Representative Report data
- National Center for Charitable Statistics (NCCS)
- 501(c)3 Charities, Scope ‘PZ’ data
All files are translated to Parquet or JSONL data before being uploaded into Data Refinery to ensure optimal performance for queries and reducing costs for users.
Reach out to a Customer Service Lead at Kingland if there is any interest in using prebuilt Background Tasks in Data Refinery Designer!