Working with the API

This page will detail how Data Refinery uses APIs and how to authenticate while working with the API.

Data Refinery uses a REST API strategy, where each API is documented using an OpenAPI annotation. The OpenAPI specification is hosted on the /swagger/index.html endpoint of the solution. This means it can be rendered in a browser by navigating to https://example.kingland-data-refinery.com/swagger/index.html.

For this document, all REST requests will be demonstrated using curl, which is a standard utility available with most Linux distributions. If curl is not available, other free tools such as Postman could be used instead.

All responses from the Data Refinery API will use application/json. Most of the requests will also accept application/json, with the exception of uploading data that uses multi-part form data to upload files.

How to Authenticate

All REST APIs for Data Refinery use an OAuth Bearer token from a user with the appropriate permissions for authentication. The /login endpoint is used to retrieve a bearer token for a user. The bearer token is stateless, and good for a limited duration. The duration is configurable per environment.

To call the API for a bearer token, use the following example:

curl https://example.kingland-data-refinery.com/login -X POST -H "Content-Type: application/json" \
  -d '{"username": "exampleUsername", "password":"examplePassword"}'

This will return a response that looks like the following:

{"code":200,"expire":"2023-06-26T17:00:00Z","token":"eya09b2hlughlkjbylasduy"}

Note. The token return in the example above has been truncated for readability reasons. A real token would be much longer.

The value within the token key is the bearer token that is used to call the remaining APIs by passing the value in the Authorization header. An example of retrieving all Source metadata using the bearer token above and curl can be seen below:

curl https://example.kingland-data-refinery.com/api/sources -H "Content-Type: application/json" \
  -H "Authorization: Bearer eya09b2hlughlkjbylasduy"

Troubleshooting

This section contains common problems users may experience when using the Data Refinery Designer API, and steps to mitigate them.

HTTP/2 GOAWAY Response

HTTP/2 is the latest version of the HTTP Protocol, designed to enhance web performance by optimizing communication between clients and servers. When interacting with our application’s API, users should leverage HTTP/2 for improved performance and efficiency.

However, users should be aware of the potential issue of a GOAWAY frame if the HTTP/2 connection is kept open for an extended period, typically around 1 hour. The GOAWAY frame is a signal from the server to the client indicating that the connection is being terminated gracefully. To prevent this issue, users should implement connection health checks and reconnect mechanisms in their applications. These mechanisms can periodically check the connection status and establish a new connection if necessary, ensuring uninterrupted communication with the API.