Cloud APIs
/
/
/
Status Processor
Last updated

Table of Contents

Introduction

The Status Processor Service allows you to monitor the status of files, data, and records ingested through the File Service, Storage API, and pecific DOMS. It is consumed by dependent services. The status can show you whether the dataflow has finished or not, and if it was successful or failed. To track the status of a dataflow, use the provided APIs.

Status Processor endpoints

The Status Processor service supports the operations listed below via different endpoints:

Query Dataset details

You can query dataset details using the GET /dataset-details endpoint based on the following parameters:

AttributeUseExampleData typeRequired (Yes/No)
correlationIdCorrelationId is a unique GUID used to capture a single request going through different stages of the Data Platform. You must send it as correlation-id header while initiating the call on the Data Platform. For File Service, the correlation-id is provided for the Metadata API call. The correlation-id is created or generated by the user, so the user already has it.0a231229-cf4b-4a71-8f7d-8a499289364dStringYes. CorrelationId or datasetId is required.
datasetIdDatasetId is a metadata record id that is returned in the response by the Metadata API while the metadata record is created. You can use the same id for querying the DataSet API of the Status Processor to fetch the correlation-id associated with this request.partitionId:dataset--File.Generic:fe57d55d-1736-42d4-b13c-115bc5f96eadStringYes. CorrelationId or datasetId is required.

Query Status

You can query the status using the POST /status/query endpoint that is available in the API Reference

Sample Status Query Request

{
  "statusQuery": {
    "correlationId": "1cc462e9-9a39-48b9-84c3-9d53b18b8089"
  },
  "limit": 10,
  "cursor": "eyJjb21wb3NpdGVUb2tlbiI6IntcInRva2VuXCI6bnVsbCxcInJhbmdlXCI6e1wibWluXCI6XCJcIixcIm1heFwiOlwiMDVDMURGRkZGRkZGRkNcIn19Iiwib3JkZXJCeUl0ZW1zIjpbeyJwcm9wZXJ0eUJhZyI6eyJpdGVtIjoxNjMwMDQ0ODI0fSwiaXRlbSI6MTYzMDA0NDgyNCwibWFwIjp7Iml0ZW0iOjE2MzAwNDQ4MjR9fV0sInJpZCI6InFHMDVBTllhK3VoNHRuVUFBQUFBQUE9PSIsImluY2x1c2l2ZSI6dHJ1ZX0"
}
{
  "statusQuery": {
    "correlationId": "1cc462e9-9a39-48b9-84c3-9d53b18b8089",
    "stage": [
      "FILE_SYNC",
      "STORAGE_SYNC"
    ],
    "status": [
      "SUCCESS"
    ]
  },
  "limit": 100
}
{
  "statusQuery": {
    "correlationId": "1cc462e9-9a39-48b9-84c3-9d53b18b8089",
    "stage": [
      "WKS_SYNC"
    ],
    "status": [
      "SKIPPED",
      "FAILED"
    ]
  },
  "limit": 100
}
{
  "statusQuery": {
    "correlationId": "1cc462e9-9a39-48b9-84c3-9d53b18b8089",
    "status": [
      "FAILED"
    ],
    "errorCode": [
      400,
      500
    ],
    "fromTimestamp": 1628571397,
    "toTimestamp": 1628639797
  },
  "limit": 100
}

Sample Status Query Request with additionalProperties:

Additional Properties is a map of key-value pairs that provides additional information for the status.

It accepts key-value pair at the first level only.

The supported data type for the keys is String and any string value can be key. Examples: inputRecordId, version, timestamp.

The supported data types for the values are String, Integer, Number, and Boolean. Example: opendes:wellbore:osdudemo-5noUsKs81vWsu0w3, true, 1234567.

You can query additional status properties using additionalProperties by specifying one or more properties.

Status RecordStatus Query

{
    "kind": "status",
    "properties": {
      "correlationId": "1cc462e9-9a39-48b9-84c3-9d53b18b8089",
      "recordId": "opendes:wellbore:wks-35b7eecfa2c35145053b150f47461bea630a.osdu.wks.wellbore.2",
      "recordIdVersion": "opendes:wellbore:wks-35b7eecfa2c35145053b150f47461bea630a.osdu.wks.wellbore.2",
      "stage": "WKS_SYNC",
      "status": "SUCCESS",
      "errorCode": 0,
      "userEmail": "test@email.com",
      "timestamp": 1625221800,
      "additionalProperties": {
        "inputRecordId": "opendes:wellbore:osdudemo-5noUsKs81vWsu0w3",
        "inputRecordIdVersion": "opendes:wellbore:osdudemo-5noUsKs81vWsu0w3:23423432"
      }
    }
  }


{
  "statusQuery": {
    "correlationId": "1cc462e9-9a39-48b9-84c3-9d53b18b8089",
    "additionalProperties": {
       "inputRecordId": "opendes:wellbore:osdudemo-5noUsKs81vWsu0w3",
       "inputRecordIdVersion": "opendes:wellbore:osdudemo-5noUsKs81vWsu0w3:23423432"
    }
  },
  "limit": 10
}


{
    "kind": "status",
    "properties": {
      "correlationId": "1cc462e9-9a39-48b9-84c3-9d53b18b8089",
      "recordId": "opendes:wellbore:wks-35b7eecfa2c35145053b150f47461bea630a.osdu.wks.wellbore.2",
      "recordIdVersion": "opendes:wellbore:wks-35b7eecfa2c35145053b150f47461bea630a.osdu.wks.wellbore.2",
      "stage": "WKS_SYNC",
      "status": "SUCCESS",
      "errorCode": 0,
      "userEmail": "test@email.com",
      "timestamp": 1625221800,
      "additionalProperties": {
        "inputRecordId": "opendes:wellbore:osdudemo-5noUsKs81vWsu0w3",
        "inputRecordIdVersion": "opendes:wellbore:osdudemo-5noUsKs81vWsu0w3:23423432"
      }
    }
  }


{
  "statusQuery": {
    "correlationId": "1cc462e9-9a39-48b9-84c3-9d53b18b8089",
    "stage": [
      "FILE_SYNC",
      "WKS_SYNC"
    ],
    "status": [
      "SUCCESS"
    ],
    "additionalProperties": {
       "inputRecordIdVersion": "opendes:wellbore:osdudemo-5noUsKs81vWsu0w3:23423432"
    }
  },
  "limit": 100
}

Publish Status

Note: Endpoint is DEPRECATED. To publish Status refer Status Publisher tutorial

You can use the POST /status endpoint to publish the Status message to statuschangedtopic. The endpoint accepts s list of statuses.

Sample Status Publish Request

[
  {
    "correlationId": "1cc462e9-9a39-48b9-84c3-9d53b18b8089",
    "recordId": "opendes:wellbore:osdudemo-ATVMxMDEzTVMxMDQ",
    "recordIdVersion": "opendes:wellbore:osdudemo-ATVMxMDEzTVMxMDQ:23423432",
    "stage": "FILE_SYNC",
    "status": "SUCCESS",
    "message": "File metadata stored successfully",
    "errorCode": 0
  }
]
[
  {
    "correlationId": "1cc462e9-9a39-48b9-84c3-9d53b18b8089",
    "recordId": "opendes:wellbore:5noUsKs81vWsu0w3",
    "stage": "WKS_SYNC",
    "status": "FAILED",
    "message": "Raw record not present",
    "errorCode": 400
  }
]

Sample Status Publish Request with additionalProperties

Additional Properties ('additionalProperties') is a map of key-value pairs that provides additional information for the status.

It accepts a key-value pair at 1st level only.

The supported data type for the keys is String and any string value can be a key. Examples: inputRecordId, version, timestamp.

Supported data types for the values are String, Integer, Long, Double, and Boolean. Examples: opendes:wellbore:osdudemo-5noUsKs81vWsu0w3, true, 1234567.

[
  {
    "kind": "status",
    "properties": {
      "correlationId": "1cc462e9-9a39-48b9-84c3-9d53b18b8089",
      "recordId": "opendes:wellbore:wks-35b7eecfa2c35145053b150f47461bea630a.osdu.wks.wellbore.2",
      "recordIdVersion": "opendes:wellbore:wks-35b7eecfa2c35145053b150f47461bea630a.osdu.wks.wellbore.2",
      "stage": "WKS_SYNC",
      "status": "SUCCESS",
      "errorCode": 0,
      "userEmail": "test@email.com",
      "timestamp": 1625221800,
      "additionalProperties": {
        "inputRecordId": "opendes:wellbore:osdudemo-5noUsKs81vWsu0w3",
        "inputRecordIdVersion": "opendes:wellbore:osdudemo-5noUsKs81vWsu0w3:23423432"
      }
    }
  }
]

More details about supported Stages and Statuses can be found in Global Status Monitoring tutorial in the Supported Stages and Statuses section.

API Specs

All available Status Processor service APIs are in the API Reference.

Global Status Monitoring

Status Publisher