• Ingestion Workflow using wellbore ingestion DAG
  • Introduction
  • Ingestion Framework
    • Components
    • Access Controls
  • How to execute the wellbore ingestion workflow
    • 1. Get Signed URL using File Service
    • 2. Upload file using signed URL
    • 3. Create File Metadata record using File Service
    • 4. Run ingestion Workflow
    • 5. Monitor Workflow Status
    • 6. Validate Ingested Data
  • File Metadata record
  • WellLog Ingestion
    • Created WellLog records
  • Trajectory Ingestion
    • Created Wellbore Trajectory records
  • Pre-registered DAG
  • Limitations
  • Known Issues

This document guides users who want to leverage the Wellbore Ingestion Directed Acyclic Graph (DAG) and have their data ingested from LAS, DLIS or LIS files. It covers the existing DAG capability, describes how to push the input LAS, DLIS or LIS files for ingestion, defines a way to trigger the DAG, how to monitor the status of ingestion and list created records.

Managed Planning Data Foundation R3 ingestion framework is built to ingest various types of data files like CSV/LAS/DLIS/LIS etc. with Apache Airflow, an open-source platform to Author, Schedule and Monitor workflows. Workflows in Airflow are collections of tasks that have directional dependencies. Specifically, Airflow uses a DAG to represent a workflow. At a high level, a DAG can be thought of as a container that holds tasks and their dependencies, and sets the context for when and how those tasks should be executed. There are various components in the framework which enable uploading a file, creating the required schemas for ingestion or invoking the ingestors. Each is explained in detail in sections that follow. The ingestion framework comes with some pre-registered DAGs. These are available out of the box and can be used by end users to execute specific workflows. Some of these pre-registered DAGs are for CSV, LAS and Document ingestion workflows.

• File Service – Facilitates management of files on data platform. Uploading, secure discovery and downloading of files are capabilities provided by File Service. API Reference

• Storage Service – JSON object store that facilitates storage of metadata information for domain entities. Also raises storage events when records are saved using Storage Service. API Reference

• Wellbore DMS – Stores curves bulk data for WellLogs. Also provides many features for Wellbore related data like Family recognition.

• Notification Service – Subscriber to storage event. Consumer interested in storage events can leverage Notification Service. Consumer should use the Register Service to register the subscription.

• Workflow Service – Facilitates management of workflows on data platform. Wrapper over the workflow engine and abstracts lot of technical nuances of the workflow engine from consumers. API Reference

• Airflow engine – Heart of the ingestion framework. Actual Workflow orchestrator.

• DAGs – Based on Direct Acyclic Graph concept, represent workflows that are authored, orchestrated, managed and monitored by the workflow engine.

  

• Access Required:

  #	Service level group	Service API Operations
  1	service.workflow.viewer	List Custom Operators
  2	service.workflow.creator	Register DAG
  3	service.workflow.admin	Register Custom Operator, Trigger Workflow, Data sharing between two tasks in ONE workflow (SAS URL)
  4	service.file.viewers & service.file.editors	To upload and download file and create File meta Data
  5	service.legal.editor & service.legal.user	Create and Retrieve Legal Tags
  6	service.schema-service.editors, service.schema-service.viewers & service.schema-service.admin	To create, update and view the schema
  7	service.storage.admin, service.storage.creator & service.storage.viewer	To create, update and view records in storage
  8	service.search.user & service.search.admin	To search records

A signed URL is required in order to be able to upload the LAS, DLIS or LIS file to cloud storage.

• API Reference

• GET Request: /file/v2/files/uploadURL

• Response:

  Sample Response

      {
          "FileID": "0bc2ad23cecb4275916089280da336ca",
          "Location": {
              "SignedURL": "https://osdur3mvpdp1qadizpdata.blob.core.windows.net/file-staging-area/osdu-user%2F1611218353490-2021-01-21-08-39-13-490%2F0bc2ad23cecb4275916089280da336ca?sv=2019-12-12&se=2021-01-28T08%3A39%3A14Z&sr=b&sp=cw&sig=y3%2FVHfeCyotI5bJU9446vHnpbIIchtrsB40szoj0QMM%3D",
              "FileSource": "/osdu-user/1611218353490-2021-01-21-08-39-13-490/0bc2ad23cecb4275916089280da336ca"
          }
      }

  • We will refer to this response in next steps. The SignedURL attribute will be used to upload the file in next step (2) and the FileSource will be used in step 4 in order to run workflow.

Upload the file using the signed URL obtained from previous step. Use a PUT request passing the input file. The file gets uploaded to cloud storage.

• PUT Request: Use the SignedURL value from previous step
• HTTP Headers:
  • 'x-ms-blob-type: BlockBlob'
• Execute the PUT request by adding the file as a binary request.

Use File Service to create metadata. The metadata enables discovery of file and secure downloads. It also provides a mechanism to query information associated with the file that is needed during its processing. Refer to File Metadata Record Reference below for more details.

• API Reference
• File Service API
• POST Request: /file/v2/files/metadata

  • Payload:

    Set the metadata data.Filesource parameter to be the FileSource value obtained from step 1.

    Sample File Generic ingestion request

        {
            "kind": "<authority>:wks:dataset--File.Generic:1.0.0",
            "data": {
                "DatasetProperties": {
                    "FileSourceInfo": {
                        "FileSource": "/osdu-user/1642503396962-2022-01-18-10-56-36-962/9c8494f978da4a289c8b959f97dedc1d",
                        "Name": "example.las"
                    }
                },
                "Endian": "LITTLE",
                "ExtensionProperties": {
                    "FileContentsDetails": {
                        "Authority": "osdu",
                        "FileType": "las"
                    }
                },
                "Name": "example.las"        
            },
            "acl": {
                "viewers": [
                    "data.default.viewers@<domain>.com"
                ],
                "owners": [
                    "data.default.viewers@<domain>.com"
                ]
            },
            "legal": {
                "legaltags": [
                    "<valid legal tag>"
                ],
                "otherRelevantDataCountries": [
                    "US"
                ],
                "status": "compliant"
            }
        }

    Sample Response

    {
    "id": "<authority>:file:4cedbed0-837b-4797-9683-f088a7a15014"
    }            

This is an explicit way to trigger the execution of an ingestion workflow using the Workflow Service workflowRun endpoint.

• API Reference: Workflow Service

• POST Request: /workflow/<workflow_name>/workflowRun

• wellbore_ingestion_wf is the default wellbore ingestion workflow_name defined by the ingestion framework. See all standard wellbore ingestion workflow names here. Create custom workflows using the Workflow service.

• HTTP Headers:

  • 'Correlation-ID': <valid correlation id>': correlation ID optional header can be set as a UUID generated by the user. This correlation ID can be used at a later stage to monitor ingestion and list created records. Another option is to get Correlation-ID header added automatically in the response.

• Payload:

  Sample Request Payload

  {
    "executionContext": {
      "id": "<file metadata id obtained in step 3 >",
      "dataPartitionId": "<dataPartition>"
    },
    "runId": "Optional. Run ID can be set explicitly by the user. If not set, the system will generate a run ID for the workflow run."
  }

  Sample Response Payload

  {
      "workflowId": "wellbore_ingestion_wf",
      "runId": "06f9c92f-e302-4987-9c69-a62abcc7e0cd",
      "startTimeStamp": 1617191215379,
      "status": "submitted",
      "submittedBy": "<user>@<domain>.com"
  }

  Error Codes

  
    | Code | Message                                       |
    | ---- | --------------------------------------------- |
    | 400  | Bad Request. Partition ID is missing/invalid. |
    | 401  | Unknown or Invalid user.                      |
    | 403  | User forbidden from accessing this API.       |
    | 404  | Not Found. Workflow Doesn't exist.            |
    

We can monitor the execution status of the workflow using the monitor Workflow Status API. The workflow run may have these possible status codes:

After the execution of a workflow run has started, monitor the execution status of the workflow run using the Workflow Service. From step 4, use the same workflow_name and the runId from the response.

• API Reference: Workflow Service

• GET Request: /workflow/<workflow_name>/workflowRun/<runId>

  Sample Response Payload

  {
      "workflowId": "wellbore_ingestion_wf",
      "runId": "06f9c92f-e302-4987-9c69-a62abcc7e0cd",
      "startTimeStamp": 1614579997265,
      "endTimeStamp": 1614607532205,
      "status": "success",
      "submittedBy": "<user>@<domain>.com",
  }

The user can find the ingested records using Status Processor Service POST /v1/status/query endpoint. The user can find the correlation Id in the Response header of the triggered workflow run (see part 4). The correlationId is a mandatory parameter using which the user can find the details of the ingested records.

• API Reference

• POST Request: /v1/status/query

• Payload:

  Sample Request Payload

  {
      "statusQuery": {
          "correlationId": "50ca4a66-ba9c-4e45-ad66-be193dac8441",
          "stage": [
             "INGESTOR"
          ],    
          "limit": 100,
          "offset": 0
      }
  }

  Sample Response Payload

  {
      "results": [
          {
          "correlationId": "50ca4a66-ba9c-4e45-ad66-be193dac8441",
          "recordId": "fc4279dc-8e65-4c82-bff8-cf7a677a69c8",
          "recordIdVersion": "1",
          "stage": "INGESTOR",
          "status": "SUBMITTED",
          "message": "Workflow run submitted. ",
          "errorCode": 0,
          "userEmail": "user@slb.com",
          "timestamp": 1629375242377
          },
          {
          "correlationId": "50ca4a66-ba9c-4e45-ad66-be193dac8441",
          "recordId": "fc4279dc-8e65-4c82-bff8-cf7a677a69c8",
          "recordIdVersion": "1",
          "stage": "INGESTOR",
          "status": "IN_PROGRESS",
          "message": "Workflow execution started",
          "errorCode": 0,
          "timestamp": 1629375258
          },
          {
          "correlationId": "50ca4a66-ba9c-4e45-ad66-be193dac8441",
          "recordId": "fc4279dc-8e65-4c82-bff8-cf7a677a69c8",
          "recordIdVersion": "1",
          "stage": "INGESTOR",
          "status": "SUCCESS",
          "message": "Workflow execution successful",
          "errorCode": 0,
          "timestamp": 1629375386
          }
      ],
      "count": 3,
      "totalCount": 3,
      "limit": 100,
      "offset": 0
  }

  • Status of records:

    Status of records for Payload

    
        {
        "statusQuery": {
            "correlationId": "50ca4a66-ba9c-4e45-ad66-be193dac8441",
            "stage": [
            "INGESTOR_SYNC"
            ]
        },
        "limit": 100,
        "offset": 0
        }

    Status of records for WellLog Ingestion Response

        {
        "results": [
            {
            "correlationId": "50ca4a66-ba9c-4e45-ad66-be193dac8441",
            "recordId": "opendes:master-data--Well:d97d4fec301d48e7bdc37763cb7ebc54",
            "stage": "INGESTOR_SYNC",
            "status": "SUCCESS",
            "errorCode": 0,
            "timestamp": 1629375336035
            },
            {
            "correlationId": "50ca4a66-ba9c-4e45-ad66-be193dac8441",
            "recordId": "opendes:master-data--Wellbore:b2641a34b46e47b38ffb8e5db34c2392",
            "stage": "INGESTOR_SYNC",
            "status": "SUCCESS",
            "errorCode": 0,
            "timestamp": 1629375336035
            },
            {
            "correlationId": "50ca4a66-ba9c-4e45-ad66-be193dac8441",
            "recordId": "opendes:work-product-component--WellLog:3dd82790207d491aac5505cf029797db",
            "stage": "INGESTOR_SYNC",
            "status": "SUCCESS",
            "errorCode": 0,
            "timestamp": 1629375336035
            }
        ]
        "count": 100,
        "totalCount": 286,
        "limit": 100,
        "offset": 0
        }

    Status of records for Wellbore Trajectory Ingestion Response

        {
        "results": [                          
            {
            "correlationId": "50ca4a66-ba9c-4e45-ad66-be193dac8441",
            "recordId": "opendes:work-product-component--WellboreTrajectory:3dd82790207d491aac5505cf029797db",
            "stage": "INGESTOR_SYNC",
            "status": "SUCCESS",
            "errorCode": 0,
            "timestamp": 1629375336035
            }
        ]
        "count": 100,
        "totalCount": 286,
        "limit": 100,
        "offset": 0
        }

Validate ingested data using the Wellbore DMS Service.

• Check Wellbore record:

  • API Reference
  • /wellbores/{record_id}
  • GET Request: /wellbores/<record_id>
  • Parameter record_id is the id of the ingested wellbore record received, for example in the status monitoring request above .
  • The response shows the ingested data.

• Check WellLog record:

  • API Reference
  • /welllogs/{record_id}
  • GET Request: /welllogs/<record_id>
  • Parameter record_id is the id of the ingested welllog record received, for example in the status monitoring request above .
  • The response shows the ingested data.

• Check WellboreTrajectory record:

  • API Reference
  • /wellboretrajectories/{record_id}
  • GET Request: /wellboretrajectories/<record_id>
  • Parameter record_id is the id of the ingested wellbore trajectory record received, for example in the status monitoring request above .
  • The response shows the ingested data.

The file Metadata is used to provide the ACL, Legal Tags, file source of the ingested data and optional attributes like custom CRS or reusing an existing Wellbore.

Sample Metadata:

{
    "acl": {
        "viewers": [
        "data.default.viewers@<domain>.com"
        ],
        "owners": [
        "data.default.viewers@<domain>.com"
        ]
    },
    "legal": {
        "legaltags": [
        "<valid legal tag>"
        ],
        "otherRelevantDataCountries": [
        "US"
        ],
        "status": "compliant"
    },
    "kind": "<authority>:wks:dataset--File.Generic:1.0.0",
    "data": {
        "DatasetProperties": {
            "FileSourceInfo": {
                "FileSource": "/osdu-user/1642523046165-2022-01-18-16-24-06-165/180f41d44aa343779f6cc1cff03f2ab8",
                "Name": "926408_100000295090_1.dlis"
            }
        },
        "ExistenceKind": "<data_partition_id>:reference-data--ExistenceKind:Prototype:", 
        "ResourceSecurityClassification": "<data_partition_id>:reference-data--ResourceSecurityClassification:RESTRICTED:",
        "Endian": "LITTLE",
        "ExtensionProperties": {
            "FileContentsDetails": {
                "Authority": "osdu",
                "FileType": "dlis",
                "FrameOfReference": [
                    {
                        "kind": "CRS",
                        "name": "MAGNA_Colombia_Bogota",
                        "persistableReference": "{\"lateBoundCRS\":{\"wkt\":\"PROJCS[\\\"MAGNA_Colombia_Bogota\\\",GEOGCS[\\\"GCS_MAGNA\\\",DATUM[\\\"D_MAGNA\\\",SPHEROID[\\\"GRS_1980\\\",6378137.0,298.257222101]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],PROJECTION[\\\"Transverse_Mercator\\\"],PARAMETER[\\\"False_Easting\\\",1000000.0],PARAMETER[\\\"False_Northing\\\",1000000.0],PARAMETER[\\\"Central_Meridian\\\",-74.0775079166667],PARAMETER[\\\"Scale_Factor\\\",1.0],PARAMETER[\\\"Latitude_Of_Origin\\\",4.59620041666667],UNIT[\\\"Meter\\\",1.0],AUTHORITY[\\\"EPSG\\\",3116]]\",\"ver\":\"PE_10_3_1\",\"name\":\"MAGNA_Colombia_Bogota\",\"authCode\":{\"auth\":\"EPSG\",\"code\":\"3116\"},\"type\":\"LBC\"},\"singleCT\":{\"wkt\":\"GEOGTRAN[\\\"MAGNA_To_WGS_1984_1\\\",GEOGCS[\\\"GCS_MAGNA\\\",DATUM[\\\"D_MAGNA\\\",SPHEROID[\\\"GRS_1980\\\",6378137.0,298.257222101]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],GEOGCS[\\\"GCS_WGS_1984\\\",DATUM[\\\"D_WGS_1984\\\",SPHEROID[\\\"WGS_1984\\\",6378137.0,298.257223563]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],METHOD[\\\"Geocentric_Translation\\\"],PARAMETER[\\\"X_Axis_Translation\\\",0.0],PARAMETER[\\\"Y_Axis_Translation\\\",0.0],PARAMETER[\\\"Z_Axis_Translation\\\",0.0],AUTHORITY[\\\"EPSG\\\",15738]]\",\"ver\":\"PE_10_3_1\",\"name\":\"MAGNA_To_WGS_1984_1\",\"authCode\":{\"auth\":\"EPSG\",\"code\":\"15738\"},\"type\":\"ST\"},\"ver\":\"PE_10_3_1\",\"name\":\"MAGNA-SIRGAS * EPSG / Colombia Bogota zone [3116,15738]\",\"authCode\":{\"auth\":\"SLB\",\"code\":\"3116001\"},\"type\":\"EBC\"}"
                    }
                ],
                "CoordinateReferenceSystemID": "<data_partition_id>:reference-data--CoordinateReferenceSystem:8cfcd4e5-9fda-5a85-93cd-4562b92e6172",
                "WellboreID": "<data_partition_id>:master-data--Wellbore:ee8d18d487ee449c9d81d0e547bae31d",
                "WellLogTypeID": "<data_partition_id>:reference-data--LogType:Raw:"
            }
        },
        "Name": "926408_100000295090_1.dlis"
    }
}

• ACL:

  • Specifies the Access Control list

  • API Reference

  • Example:

          "acl": {
              "viewers": [
                  "data.default.viewers@<domain>.com"
              ],
              "owners": [
                  "data.default.viewers@<domain>.com"
              ]
          }

• Legal Tags:

  • Specifies the legal tags. This holds the details of the data compliance for the data ingested

  • API Reference

  • Example:

      "legal": {
          "legaltags": [
              "opendes-public-usa-dataset-7643990"
          ],
          "otherRelevantDataCountries": [
              "US"
          ],
          "status": "compliant"
      }
    

• File source:

  • This is the relative file path of the LAS, DLIS, LIS, CSV, XLS, XLSX file uploaded in the file-staging-area.
  • This values is the relative file path that you have received from Get Signed Url response (mentioned above in Step 1)

• File type:

  • This is the type of file: either las, dlis, lis, csv xlsx or xls.

  • Example:

      "data": {
          "ExtensionProperties": {
              "FileContentsDetails": {
                  "FileType": "las"
              }
          }
      }
    

• Authority:

  • This attribute indicates that ingestion should use OSDU schemas. It is there for backward compatibility reasons and should always be osdu.

  • Example:

     "data": {
         "ExtensionProperties": {
             "FileContentsDetails": {
                 "Authority": "osdu"
             }
         }
     }
    

• Parent Wellbore for WellLog:

  • An existing wellbore can be provided (optional) that will be used as parent of WellLog records that will be created. The passed record needs to be a valid master-data--Wellbore. In that case, ingestion will not create any wellbore and all wellbore properties in the ingested file will not be stored anywhere (this behavior will change at a later stage).

• Parent Wellbore for Wellbore Trajectory:

  • An existing wellbore has to be provided (mandatory) that will be used as parent of WellboreTrajectory record that will be created. The parent wellbore needs to be a valid master-data--Wellbore. An invalid wellbore fails the ingestion.

  • Example:

      "data": {
          "ExtensionProperties": {
              "FileContentsDetails": {
                  "WellboreID": "opendes:master-data--Wellbore:ee8d18d487ee449c9d81d0e547bae31d"
                  }
    

• Frame of Reference (optional):

  • It is possible to explicitly provide a CRS by using this property, for example when the CRS name in the file is missing or does not match with any CRS in CRS Catalog. The CRS from the file will be ignored in that case. (refer to Created WellLog records for more details on default behavior). The data.SpatialLocation.AsIngestedCoordinates section of the parent Wellbore record is set with CRS input information.

  • Example:

      "data": {
          "ExtensionProperties": {
              "FileContentsDetails": {
                  "FrameOfReference": [
                      {
                          "kind": "CRS",
                          "name": "MAGNA_Colombia_Bogota",
                          "persistableReference": "{\"lateBoundCRS\":{\"wkt\":\"PROJCS[\\\"MAGNA_Colombia_Bogota\\\",GEOGCS[\\\"GCS_MAGNA\\\",DATUM[\\\"D_MAGNA\\\",SPHEROID[\\\"GRS_1980\\\",6378137.0,298.257222101]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],PROJECTION[\\\"Transverse_Mercator\\\"],PARAMETER[\\\"False_Easting\\\",1000000.0],PARAMETER[\\\"False_Northing\\\",1000000.0],PARAMETER[\\\"Central_Meridian\\\",-74.0775079166667],PARAMETER[\\\"Scale_Factor\\\",1.0],PARAMETER[\\\"Latitude_Of_Origin\\\",4.59620041666667],UNIT[\\\"Meter\\\",1.0],AUTHORITY[\\\"EPSG\\\",3116]]\",\"ver\":\"PE_10_3_1\",\"name\":\"MAGNA_Colombia_Bogota\",\"authCode\":{\"auth\":\"EPSG\",\"code\":\"3116\"},\"type\":\"LBC\"},\"singleCT\":{\"wkt\":\"GEOGTRAN[\\\"MAGNA_To_WGS_1984_1\\\",GEOGCS[\\\"GCS_MAGNA\\\",DATUM[\\\"D_MAGNA\\\",SPHEROID[\\\"GRS_1980\\\",6378137.0,298.257222101]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],GEOGCS[\\\"GCS_WGS_1984\\\",DATUM[\\\"D_WGS_1984\\\",SPHEROID[\\\"WGS_1984\\\",6378137.0,298.257223563]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],METHOD[\\\"Geocentric_Translation\\\"],PARAMETER[\\\"X_Axis_Translation\\\",0.0],PARAMETER[\\\"Y_Axis_Translation\\\",0.0],PARAMETER[\\\"Z_Axis_Translation\\\",0.0],AUTHORITY[\\\"EPSG\\\",15738]]\",\"ver\":\"PE_10_3_1\",\"name\":\"MAGNA_To_WGS_1984_1\",\"authCode\":{\"auth\":\"EPSG\",\"code\":\"15738\"},\"type\":\"ST\"},\"ver\":\"PE_10_3_1\",\"name\":\"MAGNA-SIRGAS * EPSG / Colombia Bogota zone [3116,15738]\",\"authCode\":{\"auth\":\"SLB\",\"code\":\"3116001\"},\"type\":\"EBC\"}"
                      }
                  ]
              }
          }
      }
    

• CoordinateReferenceSystemID(optional):

  • CoordinateReferenceSystemID allows to pass explicitely an OSDU Coordinate Reference System id into the payload of the file record for created Wellbore when ingesting WellLog and for Wellbore Trajectory. Its value can be set to the data.projectedCrsID attribute of the created WellboreTrajectory record. If there is no CoordinateReferenceSystemID passed into the payload then the service tries to resolve from the Trajectory header file a CRS name and its corresponding CoordinateReferenceSystemID in the CoordinateReferenceSystem OSDU catalog to be set to the data.projectedCrsID attribute of the WellboreTrajectory record.

  • Pattern to be respected : "^[\\w\\-\\.]+:reference-data\\-\\-CoordinateReferenceSystem:[\\w\\-\\.\\:\\%]+:[0-9]*$:"

  • Example:

      "data": {
          "CoordinateReferenceSystem": "opendes:reference-data--CoordinateReferenceSystem:WGS_1984_UTM_Zone_28S:"
      }
    

• ExistenceKind(optional):

  • ExistenceKind describes at which stage the instance is in the cradle-to-grave span of its existence. Typical values are: "Proposed", "Planned", "Active", "Inactive", "Marked for Disposal", "Prototype". This is an attribute to be added to Well, Wellbore, WellLog and Wellbore Trajectory metadata.

  • Example:

      "data": {
          "ExistenceKind": "opendes:reference-data--ExistenceKind:Prototype:"
      }
    

• ResourceSecurityClassification(optional):

  • This is used to describe the resource security classifications. Typical values are: "Confidential", "Private", "Public", "Secret", "Restricted". This is an attribute to be added to Well, Wellbore, WellLog and Wellbore Trajectory metadata.

  • Example:

      "data": {
          "ResourceSecurityClassification": "opendes:reference-data--ResourceSecurityClassification:RESTRICTED:"
      }
    

• WellLogTypeID(optional):

  • This is used to describe the type of logs. Typical values are: "Edited", "Interpreted", "Raw", "Corrected". This is an attribute to be added to WellLog metadata.

  • Example:

      "data": {
          "ExtensionProperties": {
              "FileContentsDetails": {
                  "WellLogTypeID": "opendes:reference-data--LogType:Raw:"
              }
          }
      }
    

WellLog Ingestion supports 2 versions of WellLog, which are: osdu:wks:work-product-component--WellLog:1.1.0 and osdu:wks:work-product-component--WellLog:1.2.0.

osdu:wks:work-product-component--WellLog:1.1.0 is the default ingestion target kind, it happens also when the TargetKind is set as osdu:wks:work-product-component--WellLog:1.1.0 in file record.

If the user wants to ingest a file as WellLog:1.2.0, the TargetKind in the file record needs to be set as osdu:wks:work-product-component--WellLog:1.2.0 so that Ingestion can generate the WellLog Entity based on this version of the WellLog.

Example:

"data": {
    "ExtensionProperties": {
        "FileContentsDetails": {
            "TargetKind": "osdu:wks:work-product-component--WellLog:1.2.0"
        }
    }
}
      

Supported file formats are:

• Log ASCII Standard (LAS) version 1.0, 1.2 and 2.0 files including some LAS files not compliant with the specification that mixes features from different versions and Techlog specific extensions
• Digital Log Interchange Standard (DLIS) version 1 files
• Log Information Standard (LIS79) files

The file ingestion creates records with osdu:wks:master-data--Well:1.0.0, osdu:wks:master-data--Wellbore:1.0.0, and osdu:wks:work-product-component--WellLog:1.1.0 kinds.

Users can pass the name of their database to the File.Generic payload as Source under FileContentsDetails. The data.Source attribute is set into the WellLog, Wellbore, Well through Ingestion by reading the Source parameter in the File.Generic record

• Well and Wellbore records creation:

  • The service creates both Well and Wellbore records if there is no WellboreID passed into the payload of the file generic record.

  • If a UWI is present in ingested file, it gets assigned to data.FacilityID and as an alias in data.NameAliases.

  • When data.FacilityID is added from UWI, the file ingestion also sets data.FacilityTypeID as data-partition-id:reference-data--FacilityType:Wellbore: for Wellbore record and as data-partition-id:reference-data--FacilityType:Well: for Well record.

  • It is possible to set the data.SpatialLocation.AsIngestedCoordinates.persistableReferenceCrs property of the Wellbore record:

    • Passing explicitely the CRS through the FrameOfReference option (optional).
    • If there is no CRS explicitely passed through the FrameOfReference option, the file ingestion searches for a CRS name in the header of the file (LAS, DLIS) and tries to resolve the persistableReferenceCrs.

  • Users can pass a CoordinateReferenceSystemID(optional).

    • When this CoordinateReferenceSystemID is passed explicitly, a Wellbore record is created with the CoordinateReferenceSystemID value set to the data.SpatialLocation.AsIngestedCoordinates.CoordinateReferenceSystemID property.
    • Otherwise the service tries to identify in the LAS or DLIS header file a CRS name. It searches in OSDU CoordinateReferenceSystem reference catalog for a CRS with the same name or alias and set the result to the data.SpatialLocation.AsIngestedCoordinates.CoordinateReferenceSystemID property.

  • When some elevations properties are found in the ingested file, they are added in data.VerticalMeasurements of the Wellbore record (e.g. EREF and DREF properties).

  • If there is only one elevation in the ingested file or datum, it is assigned as DefaultVerticalMeasurementID.

  • In case there are more than one elevation in the ingested file, DefaultVerticalMeasurementID is assigned depending on the datum of those elevations based on this order of priority:

    • KB (Kelly Bushing)
    • GR (Ground Level)
    • MSL (Mean Sea Level).

  • If the datum is not found in the above list, the file ingestion chooses the first datum in the file to be set as the DefaultVerticalMeasurementID.

• WellLog creation:

  • A family is assigned to each curve in the WellLog record based on their mnemonic and unit if they can be found in family catalog. Otherwise family won't be added.
  • NaN and Infinity values coming from DLIS properties are stored as string values.
  • The bulk data associated to the WellLog record is stored to Wellbore DMS and can be retrieved through the WDDMS WellLog data API.
  • First value in reference curve is assigned to data.TopMeasuredDepth and data.SamplingStart.
  • Last value in reference curve is assigned to data.BottomMeasuredDepth and data.SamplingStop.
  • data.SamplingInterval is added when sampling rate in reference curve is constant. data.IsRegular flag is also set to true in that case or set to false otherwise (for WellLog >1.2.0 only).

It happens when TargetKind is osdu:wks:work-product-component--WellboreTrajectory:1.1.0. WellboreID property in File record is mandatory.

Supported file formats are:

• CSV files
• Excel files (XLSX, XLS)

with following Meta data structure: UWI, Source, ServiceCompanyID, SurveyType, SurveyToolTypeID...

Ingestion creates a record with osdu:wks:work-product-component--WellboreTrajectory:1.1.0 kind.

A reference to a TrajectoryStationPropertyType is assigned to each station (channel, curve) in the WellboreTrajectory record with the following logic:

• Trying to resolve a family from the family catalog based on the mnemonic and unit of the channel.
• From the family and azimuth reference type (for some channels only) to find a match with code or alias name in the OSDU TrajectoryStationPropertyType reference catalog. If one of those steps fails a TrajectoryStationPropertyTypeID is assigned with theTrajectoryStationPropertyType` reference data pattern concatenated with the channel mnemonic.

If there is no vertical measurement (Elevation value and Datum of reference) defined in the header of the ingested file, the file ingestor tries to find in the parent Wellbore record the DefaultVerticalMeasurementID and to reference this default vertical measurement in the WellboreTrajectory record.

Users can pass a CoordinateReferenceSystemID(optional). When this CoordinateReferenceSystemID is passed explicitly, a WellboreTrajectory record is created with the CoordinateReferenceSystemID value set to the data.ProjectedCRSID property. Otherwise the service tries to identify in the trajectory header file a CRS name. It searches in OSDU CoordinateReferenceSystem reference catalog for a CRS with the same name or alias and set the result to the data.ProjectedCrsID property.

Users can pass the name of their database to the File.Generic payload as Source under FileContentsDetails. The data.Source attribute is set into the WellboreTrajectory through Ingestion with the following order of priority: By reading the Source parameter in the File.Generic record By reading the Source from the header file (CVX trajectory format)

The bulk data associated to the WellboreTrajectory record is stored to Wellbore DMS and can be retrieved through the WDDMS WellboreTrajectory data API.

The ingestion framework comes with pre-registered DAG(s) to do ingestion and the ingestion framework created the corresponding workflow(s), which are available to use:

• workflow name: wellbore_ingestion_wf

Ingestion Workflow Service known limitations:

• Not yet integrated with Managed Planning Data Foundation Notification Service
  • Manually call the Workflow Service to execute a registered DAG.
• Scalability limitations: 90 queued up workflow requests, with 10 concurrent workflows execution.

• N/A