The Workflow Service API is a wrapper API on the existing orchestrator services (Apache Airflow: https://airflow.apache.org/) that satisfies the domain workflow needs. You can create and maintain the workflows using this API. This API enables users to perform operational tasks such as start, stop, or pause the workflows.
A workflow is a collection of steps that enables users to perform tasks in a particular order. For example, CSV workflow performs steps needed to successfully ingest a CSV file into the data platform. Each workflow is represented by a unique workflow name. Workflows are represented as Directed Acyclic Graphs (DAGs). In case of Airflow, each workflow will correspond to an Airflow DAG.
- Public workflows: The workflows are created and open-sourced by a community in Gitlab. These workflows are deployed as a part of the Managed Planning Data Foundation. Public workflows are available across data partitions. The workflow runs of public workflow adhere to data partition boundaries. In case of SAS deployment, single public workflow is available to all the data partitions. In the case of enterprise deployments, public workflows are present across multiple data partitions of the enterprise.
- Private workflows: Private workflows are connected to one data partition. If two data partitions in one enterprise wants to have a common workflow, they both need to configure the workflow. Private workflows can be the proprietary workflows of the company.
A workflow run is a single execution of a workflow. For example, a CSV workflow executed to ingest a CSV file is a CSV workflow run. When Airflow is the orchestrator, a workflow run is equivalent to one DAG run. Each instance of a workflow run is represented by a unique run id. The status of the workflow run can be as follows:
- SUBMITTED: Workflow trigger accepted by platform but actual workflow run has not started.
- RUNNING: Workflow in progress or under execution.
- SUCCESS: Workflow run completed successfully.
- FAILED: Workflow run completed but failed.
| # | Service level group | Service API operations |
|---|---|---|
| 1 | service.workflow.viewer | List custom operators. |
| 2 | service.workflow.creator | Register DAG |
| 3 | service.workflow.admin | Register custom operator, Trigger workflow, and share data between two tasks in one workflow (SAS URL). |
A workflow admin can create an Airflow custom operator for a logic which is used across multiple workflows. The workflow admin should have access to service level group service.workflow.admin. Examples are units, CRS (Coordinate Reference System), and Relationships and StatusMonitoring operators. The following access are required:
The workflow services includes the following operations:
- Workflow management
- Workflow execution management
- Custom operators management
Workflow management includes the following endpoints:
GET /workflow: Lists all the workflows applicable for a partition.POST /workflow: Registers a workflow definition composed of standard and custom operators in the orchestration tool.GET /workflow/{workflow_name}: Lists the workflow by name.DELETE /workflow/{workflow_name}: Deletes the workflow by workflow name.
Workflow execution management includes the following endpoints:
GET /workflow/{workflow_name}/workflowRun: Lists the details of all the workflows.POST /workflow/{workflow_name}/workflowRun: Triggers a workflow execution.GET /workflow/{workflow_name}/workflowRun/{runId}: Lists the details of a specific workflow run instance.PUT /workflow/{workflow_name}/workflowRun/{runId}: Updates the status of a workflow run.GET /workflow/{workflowId}/workflowRun/{runId}/getSignedUrl: Lists the signed URL of a location that can be shared by tasks.
Custom operators management includes the following endpoints:
POST /customOperator: Add a new custom operator.GET/customOperator: Lists all the registered custom operators.GET /customOperator/{customOperatorId}: Lists the custom operator by ID.