- Ingestion Workflow using CSV DAG
This document guides users who want to leverage the pre-registered CSV Directed Acyclic Graph (DAG) and have their data ingested from CSV files. It covers the existing CSV DAG capability, defines a way to trigger the DAG and describes how to push the input CSV files for ingestion.
The OSDU R3 ingestion framework is built to ingest various data file types such as Shapefile, LAS, DLIS, and so on, with Apache Airflow, an open-source platform to author, schedule, and monitor workflows. Workflows in Airflow are collections of tasks that have directional dependencies. Specifically, Airflow uses a DAG to represent a workflow. At a high level, a DAG can be thought of as a container that holds tasks and their dependencies, and sets the context for when and how those tasks are executed. There are various components in the framework which enable uploading a file, creating the required schemas for ingestion, or invoking the ingestors. Each is explained in detail in the sections that follow. The ingestion framework comes with several preregistered DAGs. These are available out-of-the-box, and you can use them to execute specific workflows. Some of these preregistered DAGs are for shapefile, CSV, LAS, and Document ingestion workflows.
File service – This service facilitates management of files on the data platform. Uploading, secure discovery, and downloading of files are capabilities provided by File Service.
Storage service – This JSON object store facilitates storage of metadata information for domain entities. It also raises storage events when the records are saved using the Storage service.
Notification service – Consumers use this service to subscribe to storage events. Use the Register service to register the subscription.
Workflow service – This service facilitates the management of workflows on the data platform by providing a wrapper around the workflow engine to abstract some of the technical nuances of the workflow engine from consumers.
Airflow engine – This service is the heart of the ingestion framework and acts as a workflow orchestrator.
DAGs – This service is based on Direct Acyclic Graph concept and represent workflows that are authored, orchestrated, managed, and monitored by the workflow engine.
Access required:
# Service level group Service API operations 1 service.workflow.viewer List custom operators 2 service.workflow.creator Register DAG 3 service.workflow.admin Register custom operator, trigger workflow, and data sharing between two tasks in ONE workflow (SAS URL) 4 service.file.viewers & service.file.editors To upload and download the file and create file metadata 5 service.legal.editor & service.legal.user Create and retrieve legal tags 6 service.schema-service.editors, service.schema-service.viewers & service.schema-service.admin To create, update, and view the schema 7 service.storage.admin, service.storage.creator & service.storage.viewer To create, update, and view the records in storage 8 service.search.user & service.search.admin To search the records
Use the Schema service to register your schema that describes your data model. Refer to the Schema service tutorial and the API Reference
A signed URL is required in order to upload the CSV file to cloud storage.
GET Request: /file/v2/files/uploadURL
Response:
Sample Response
{ "FileID": "0bc2ad23cecb4275916089280da336ca", "Location": { "SignedURL": "https://osdur3mvpdp1qadizpdata.blob.core.windows.net/file-staging-area/osdu-user%2F1611218353490-2021-01-21-08-39-13-490%2F0bc2ad23cecb4275916089280da336ca?sv=2019-12-12&se=2021-01-28T08%3A39%3A14Z&sr=b&sp=cw&sig=y3%2FVHfeCyotI5bJU9446vHnpbIIchtrsB40szoj0QMM%3D", "FileSource": "/osdu-user/1611218353490-2021-01-21-08-39-13-490/0bc2ad23cecb4275916089280da336ca" } }- We will refer to this response in next steps. The SignedURL attribute will be used to upload the file in next step (3) and the FileSource will be used in step 5 in order to run workflow.
Upload the file using the signed URL obtained from previous step. Use a PUT request passing the input file. The file is uploaded to cloud storage.
- PUT Request: Use the SignedURL value from previous step
- HTTP Headers:
'x-ms-blob-type: BlockBlob''content-type: text/csv'
- Execute the PUT request by adding the file as a binary request.
Use the File service to create metadata. The metadata enables the discovery of the file and secure downloads. It also provides a mechanism to query information associated with the file that is required during its processing. For more information, refer to the File Metadata Record Reference below for more details.
API Reference
POST request: /file/v2/files/metadata
Payload:
Set the metadata
data.Filesourceparameter to be the FileSource value obtained from step 2.Sample Request
{ "data": { "SchemaFormatTypeID": "string", "Name": "Wellbore-Sample_data_v2.csv", "EncodingFormatTypeID": "string", "Endian": "BIG", "Checksum": "string", "DatasetProperties": { "FileSourceInfo": { "Name": "File", "PreLoadFilePath": "string", "FileSource": "/osdu-user/1626679045186-2021-07-19-07-17-25-186/b0e44c076fbf4ce0998a6af06891aed2", "PreloadFileCreateUser": "string", "PreloadFileCreateDate": "string", "PreloadFileModifyUser": "string", "PreloadFileModifyDate": "string", "FileSize": "string", "EncodingFormatTypeID": "string" } }, "ExtensionProperties": { "Classification": "Raw File", "Description": "An text further describing this file example.", "ExternalIds": [ "string" ], "FileDateCreated": {}, "FileDateModified": {}, "FileContentsDetails": { "TargetKind": "<authority>:<source>:wellbore:1.0.0", "nestedFieldDelimiter":".", "FileType": "csv", "SpatialMapping": { "type": "point", "latitude": "location.[0].latitude", "longitude": "location.[0].longitude" }, "FrameOfReference": [ { "kind": "CRS", "name": "GCS_WGS_1984", "persistableReference": "{\"wkt\":\"GEOGCS[\\\"GCS_WGS_1984\\\",DATUM[\\\"D_WGS_1984\\\",SPHEROID[\\\"WGS_1984\\\",6378137.0,298.257223563]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433],AUTHORITY[\\\"EPSG\\\",4326]]\",\"ver\":\"PE_10_3_1\",\"name\":\"GCS_WGS_1984\",\"authCode\":{\"auth\":\"EPSG\",\"code\":\"4326\"},\"type\":\"LBC\"}", "propertyNames": [ "location.[0].latitude", "location.[0].longitude", "location.[1].latitude", "location.[1].longitude" ], "propertyValues": [ "deg" ], "uncertainty": 0 }, { "kind": "DateTime", "persistableReference": "{\"type\": \"DAT\", \"format\": \"MM-dd-yyyy\"}", "name": "date", "propertyNames": [ "permit.[1]", "initial_completion_date", "status_date", "spud_date" ], "propertyValues": [ "permit.[1]", "initial_completion_date", "status_date", "Sspud_date" ], "uncertainty": 0 }, { "kind": "Unit", "name": "ft", "persistableReference": "{\"scaleOffset\":{\"scale\":0.3048,\"offset\":0.0},\"symbol\":\"ft\",\"baseMeasurement\":{\"ancestry\":\"Length\",\"type\":\"UM\"},\"type\":\"USO\"}", "propertyNames": [ "md", "tvd", "elevation" ], "propertyValues": [ "ft" ], "uncertainty": 0 } ], "relationships": [ { "project": { "ids": [ "<authority>:<source>:project-sxs0f1a5219-4640-50af-9f63:", "<authority>:<source>:project-dsv05f45665-5885-40ad-d9m3:" ] } } ], "relatedNaturalKey": [ { "well": { "targetKind": "<authority>:<source>:well:1.0.0", "keys":[ { "sourceColumn": "relationshipsToSet.parent_uwi", "targetAttribute": "UWI" } ] } }, { "states": { "targetKind": "<authority>:<source>:states:1.0.0", "keys":[ { "sourceColumn": "state", "targetAttribute": "STATE_NAME" } ] } }, { "trajectory": { "keys":[ { "sourceColumn": "parentDefTrajectory" } ] } }, { "field": { "keys":[ { "sourceColumn": "parentField" } ] } } ] } } }, "meta": [], "id": "<authority>:dataset--File.Generic:dec44cb4-916c-4bee-85da-0de713a32f26", "version": 1610634966669814, "kind": "<authority>:wks:dataset--File.Generic:1.0.0", "acl": { "viewers": [ "data.default.viewers@{domain}.com" ], "owners": [ "data.default.viewers@{domain}.com" ] }, "legal": { "legaltags": [ "<valid legal tag>" ], "otherRelevantDataCountries": [ "US" ], "status": "compliant" } }Sample Response
{ "id": "<authority>:file:4cedbed0-837b-4797-9683-f088a7a15014" }
You can trigger the execution of an ingestion workflow using the Workflow service workflowRun endpoint.
API reference: Workflow Service
POST request: /workflow/
<workflow_name>/workflowRuncsv_parser_wf_status_gsmis the default CSV ingestionworkflow_namedefined by the ingestion framework. See the list of all standard CSV workflow names here. Create custom workflows using the Workflow service.Payload:
Sample Request Payload
{ "executionContext": { "id": "<file metadata id obtained in step 4 >", "dataPartitionId": "<dataPartition>" }, "runId": "Optional. Run ID can be set explicitly by the user. If not set, the system will generate a run ID for the workflow run." }Sample Response Payload
{ "workflowId": "csv_parser_wf_status_gsm", "runId": "06f9c92f-e302-4987-9c69-a62abcc7e0cd", "startTimeStamp": 1617191215379, "status": "submitted", "submittedBy": "<user>@<domain>.com" }Error Codes
| Code | Message | | ---- | --------------------------------------------- | | 400 | Bad Request. Partition ID is missing/invalid. | | 401 | Unknown or Invalid user. | | 403 | User forbidden from accessing this API. | | 404 | Not Found. Workflow Doesn't exist. |
We can monitor the execution status of the workflow using the monitor Workflow Status API. The workflow run may have these possible status codes:
| Code | Message |
|---|---|
| SUBMITTED | Workflow trigger accepted by platform, but actual workflow run has not started. |
| RUNNING | Workflow in progress or under execution. |
| FAILED | Workflow execution completed with failures. |
| SUCCESS | Workflow execution completed successfully. |
After the execution begins, monitor the execution status of the workflow using the Workflow service. Use the same workflow_name and the runId from the response in Step 5.
API reference: Workflow Service
GET request: /workflow/
<workflow_name>/workflowRun/<runId>Sample Response Payload
{ "workflowId": "csv_parser_wf_status_gsm", "runId": "06f9c92f-e302-4987-9c69-a62abcc7e0cd", "startTimeStamp": 1614579997265, "endTimeStamp": 1614607532205, "status": "success", "submittedBy": "<user>@<domain>.com", }
Search the ingested records using the Search service POST /query_with_cursor endpoint.
POST request: /query_with_cursor
Payload:
Sample Request Payload
{ "kind": "<metadata target kind>", "query": "<search query>" }Sample Response Payload
{ "cursor": "<cursor_id>", "results": [ { "data": {}, "kind": "<kind>", "id": "<record_id>" } ], "totalCount": 1 }
You can find the ingested records using the Status Processor service POST /v1/status/query endpoint. You can find the correlation ID in the response header of the triggered workflow run. The correlationId is a mandatory parameter with which you can find the details of the ingested records.
POST request: /v1/status/query
Payload:
Sample Request Payload
{ "statusQuery": { "correlationId": "50ca4a66-ba9c-4e45-ad66-be193dac8441", "stage": [ { "INGESTOR" } ], "limit": 100, "offset": 0 }Sample Response Payload
{ "results": [ { "correlationId": "50ca4a66-ba9c-4e45-ad66-be193dac8441", "recordId": "fc4279dc-8e65-4c82-bff8-cf7a677a69c8", "recordIdVersion": "1", "stage": "INGESTOR", "status": "SUBMITTED", "message": "Workflow run submitted. ", "errorCode": 0, "userEmail": "user@slb.com", "timestamp": 1629375242377 }, { "correlationId": "50ca4a66-ba9c-4e45-ad66-be193dac8441", "recordId": "fc4279dc-8e65-4c82-bff8-cf7a677a69c8", "recordIdVersion": "1", "stage": "INGESTOR", "status": "IN_PROGRESS", "message": "Workflow execution started", "errorCode": 0, "timestamp": 1629375258 }, { "correlationId": "50ca4a66-ba9c-4e45-ad66-be193dac8441", "recordId": "fc4279dc-8e65-4c82-bff8-cf7a677a69c8", "recordIdVersion": "1", "stage": "INGESTOR", "status": "SUCCESS", "message": "Workflow execution successful", "errorCode": 0, "timestamp": 1629375386 } ], "count": 3, "totalCount": 3, "limit": 100, "offset": 0 }Status of records:
Status of records for Payload
{ "statusQuery": { "correlationId": "50ca4a66-ba9c-4e45-ad66-be193dac8441", "stage": [ "INGESTOR_SYNC" ] }, "limit": 100, "offset": 0 }Status of records for Response
{ "results": [ { "correlationId": "50ca4a66-ba9c-4e45-ad66-be193dac8441", "recordId": "opendes:DSPRaw:ProExcel26-MTI1Mzc4LjA", "recordIdVersion": "1", "stage": "INGESTOR_SYNC", "status": "FAILED", "message": "Storing failed with following error: Storing failed with following error: 400 Bad Request: [{\"code\":400,\"reason\":\"Bad request\",\"message\":\"Cannot update the same record multiple times in the same request. Id: opendes:DSPRaw:ProExcel26-MTI1Mzc4LjA\"}]", "errorCode": 400, "timestamp": 1629375336035 }, "count": 100, "totalCount": 286, "limit": 100, "offset": 0 }
| Stage | Message |
|---|---|
| INGESTOR | Job-level messages. |
| INGESTOR_SYNC | Record-level messages. |
| Status | Message |
|---|---|
| SUBMITTED | Workflow trigger accepted by the platform, but actual the workflow run has not started. |
| IN_PROGRESS | Workflow in progress or under execution. |
| FAILED | Workflow execution completed with failures. |
| SUCCESS | Workflow execution completed successfully. |
Validate ingested data using the Storage service.
- API Reference
- GET request: /storage/v2/records/
<record_id> - Parameter
record_idis the ID of the received ingested record, for example, in the response of the above search query. - The response shows the ingested data.
CSV file:
- The first row of the CSV file is the header which represents the attributes of the ingested data. To maintain full data fidelity, the CSV headers should match 'ONE to ONE' with the raw schema specified by the user.
- To support the ingestion of data from CSV files into nested schemas, you can specify a delimiter character, as described on the File Metadata record section, to define the nested level in the CSV header. For example, if the
nestedFieldDelimiterin the file metadata is set to., a nested attribute described in the CSV header should look similar toelevationReference.elevationFromMsl.unitKey. - To support the ingestion of data from CSV files into schemas with nested arrays, you can specify a delimiter character, as described in the File Metadata record section, to define the nested array levels in the CSV header. For example, if the
nestedFieldDelimiterin the file metadata is set to., a nested attribute described in the CSV header should look similar toTerms.[0].ObligationTypeID,Terms.[1].ObligationTypeID. Here[]is used to represent arrays; therefore you cannot use[]in the attribute's name. - From the second row onwards, each row represents a data record to be ingested.
- If the target schema has attributes tagged as "x-osdu-natural-key", they are used to generate the
record id. For example, a combination of UWI+WELLNAME acting as a natural key will result in an ID like<authority>:<source>:<entitytype>-{64bit encoded value of the UWI+WELLNAME defined in the CSV file}. If the target schema does not have any attributes tagged as "x-osdu-natural-key" or if the natural key attributes are missing in CSV file, the ID will be generated by the Storage service. The "x-osdu-natural-key" index must start at 1. - To establish relationships, the key attributes of the parent records should be present in the CSV file.
- This sample CSV file ingests 1 wellbore and establishes deterministic and non-deterministic relationships with the Parents field, well, states, and trajectory using the attributes mentioned in the file metadata. The wellbore.csv also contains attributes to support the ingestion of data into schemas with nested/nested arrays.
- For details refer to Relationships Reference
- Sample CSV file for wellbore and excel file explaining the structure: Wellbore.csv Wellbore_sampledata_struture.csv
Schema service
Use the Schema service to map the attributes and their kinds. The Schema kind should be unique; construct it using the attributes under schemaIdentity. The basic format is
<authority>:<source>:<entityType>:<version>A sample schema for wellbore
Sample file: Wellbore_Schema.json
{ "schemaInfo": { "schemaIdentity": { "authority": "<authority>", "source": "<source>", "entityType": "wellbore", "schemaVersionMajor": 1, "schemaVersionMinor": 0, "schemaVersionPatch": 0 }, "status": "DEVELOPMENT" }, "schema": { "license": "Copyright 2017-2020, Schlumberger\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\nhttp://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n", "$schema": "http://json-schema.org/draft-07/schema#", "description": "The wellbore schema. Used to capture the general information about a wellbore. This information is sometimes called a \"wellbore header\". A wellbore represents the path from the surface to a unique bottomhole location. The wellbore object is uniquely identified within the context of one well object.", "id": "https://slb-swt.visualstudio.com/data-management/Ingestion%20Services/_git/wke-schema?path=%2Fdomains%2Fwell%2Fjson_schema%2Fslb_wke_wellbore.json&version=GBmaster", "title": "Wellbore", "type": "object", "definitions": { "wellboreData": { "description": "The domain specific data container for a wellbore.", "title": "Wellbore Data", "type": "object", "properties": { "identification": { "description": "The identification key tags associated with this wellbore.", "title": "uniqueKey", "x-osdu-indexing": { "type": "nested" }, "$ref": "#/definitions/identification" }, "location": { "description": "Record locations", "type": "array", "x-osdu-indexing": { "type": "nested" }, "items": { "$ref": "#/definitions/location" } }, "spud_date": { "format": "date", "description": "The date and time when activities to drill the borehole begin to create a hole in the earth. For a sidetrack, this is the date kickoff when operations began. The format follows ISO 8601 YYYY-MM-DD extended format.", "x-slb-aliasProperties": [ "witsml:DTimKickoff", "ocean:SPUD_DATE", "drillplan:spud_date" ], "title": "Spud Date", "type": "string", "example": "2013-03-22" }, "md": { "x-slb-measurement": "Measured Depth", "description": "Measured Depth", "title": "Measured Depth", "type": "number", "example": 12.2 }, "tvd": { "x-slb-measurement": "True Vertical Depth", "description": "True Vertical Depth", "title": "True Vertical Depth", "type": "number", "example": [ 20711 ] }, "operator": { "description": "The operator of the wellbore.", "x-slb-aliasProperties": [ "ocean:Operator", "witsml:Operator" ], "title": "Operator", "type": "string", "example": "Anadarko Petroleum" }, "original_operator": { "description": "The original operator of the wellbore.", "x-slb-aliasProperties": [ "ocean:Operator", "witsml:Operator" ], "title": "Original Operator", "type": "string", "example": "Anadarko Petroleum" }, "elevation": { "x-slb-measurement": "Elevation", "description": "Elevation", "title": "Elevation", "type": "integer", "example": [ 84 ] }, "elevation_ref": { "description": "Elevation reference used for the measurements", "title": "Elevation reference", "type": "string", "example": "MSL" }, "lease": { "description": "The lease name, to which the wellbore belongs.", "x-slb-aliasProperties": [ "witsml:LEASE" ], "title": "LEASE", "type": "string", "example": "SMP G09995" }, "permit": { "type": "array", "title": "Issued permit number and date", "description": "The number and date/time (issue date) of the wellbore permit. The date format follows the ISO 8601 YYYY-MM-DD extended format.", "minItems": 2, "items": { "type": "string" }, "example": [ "SMP-09995", "2013-03-22" ] }, "country": { "description": "The country where the wellbore is located. The country name follows the convention in ISO 3166-1 'English short country name', see https://en.wikipedia.org/wiki/ISO_3166-1.", "x-slb-aliasProperties": [ "witsml:Country" ], "title": "Country", "type": "string", "example": [ "United States of America" ] }, "state": { "description": "The state where the wellbore is located.", "x-slb-aliasProperties": [ "witsml:State" ], "title": "State", "type": "string", "example": [ "Texas" ] }, "county": { "description": "The county where the wellbore is located.", "x-slb-aliasProperties": [ "witsml:County" ], "title": "County", "type": "string", "example": [ "ATWATER VALLEY" ] }, "field": { "description": "The field name to which the wellbore belongs.", "x-slb-aliasProperties": [ "witsml:Field" ], "title": "Field", "type": "string", "example": "ATWATER VLLY B 8" }, "basin": { "description": "The basin name to which the wellbore belongs.", "x-slb-aliasProperties": [ "witsml:BASIN" ], "title": "Basin", "type": "string", "example": "ATWATER" }, "formation_at_td": { "description": "The formation name at the wellbore total depth.", "x-slb-aliasProperties": [ "witsml:FORMATION_AT_TD" ], "title": "Formation at TD", "type": "string", "example": "MIOCENE LOWER" }, "class": { "description": "The current class of the wellbore.", "title": "class", "type": "string", "example": "NEW FIELD WILDCAT" }, "status": { "description": "The current status of the wellbore.", "title": "Status", "type": "string", "example": "DRY & ABANDONED" }, "status_date": { "format": "date", "description": "The date and time of the current status of the wellbore. The format follows the ISO 8601 YYYY-MM-DD extended format.", "x-slb-aliasProperties": [ "witsml:DTimKickoff", "ocean:STATUS_DATE", "drillplan:STATUS_DATE" ], "title": "Status Date", "type": "string", "example": "2013-03-22" }, "initial_completion_date": { "format": "date", "description": "The date and time of the initial completion of the wellbore. The format follows the ISO 8601 YYYY-MM-DD extended format.", "x-slb-aliasProperties": [ "witsml:DTimKickoff", "ocean:INITIAL_COMPLETION_DATE", "drillplan:INITIAL_COMPLETION_DATE" ], "title": "Initial Completion Date", "type": "string", "example": "2013-03-22" }, "wellbore_shape": { "description": "The shape of the wellbore.", "title": "Wellbore Shape", "type": "string", "example": [ "DIRECTIONAL", "VERTICAL" ] }, "unit_system": { "description": "Unit system used for the wellbore measurements.", "title": "Unit Sustem", "type": "string", "example": "English" }, "terms": { "description": "A list of obligations or allowed activities specified by the agreement that apply to stored resources. These are translated into rules that the Entitlement Rulebase enforces. Each rule should reference the agreement it codifies.", "x-osdu-indexing": { "type": "nested" }, "type": "array", "properties": { "startDate": { "format": "date-time", "type": "string", "description": "The date when the obligation becomes enforceable." }, "obligationDescription": { "type": "string", "description": "Lengthy description of a legal restriction imposed on data governed by the agreement." }, "obligationTypeID": { "pattern": "srn:reference-data/ObligationType:[^:]+:[0-9]*", "type": "integer", "description": "Reference to the general class of obligation, such as nondisclosure, termination of use, nonassignment, export restriction, and limitation on derivatives." }, "endDate": { "format": "date-time", "type": "string", "description": "The date when the obligation no longer needs to be fulfilled." } } }, "fieldId": { "x-osdu-relationship": [ { "EntityType": "field" } ], "pattern": "^[\\w\\-\\.]+:master-data\\-\\-field:[\\w\\-\\.\\:\\%]+:[0-9]*$", "type": "string" }, "defTrajectoryId": { "x-osdu-relationship": [ { "EntityType": "trajectory" } ], "pattern": "^[\\w\\-\\.]+:[\\w\\-\\.]+:[\\w\\-\\.]+:[\\w\\-\\.]*$", "type": "string" }, "wellId": { "x-osdu-relationship": [ { "EntityType": "well", "GroupType": "master-data" } ], "type": "string" }, "projectIds": { "description": "The relationship tags associated with this entity.", "title": "Project IDs", "type": "object", "x-osdu-indexing": { "type": "nested" }, "$ref": "#/definitions/parentProjectIds" }, "relationshipsToSet": { "description": "The relationship tags associated with this entity.", "title": "relationships", "x-osdu-indexing": { "type": "nested" }, "$ref": "#/definitions/relationshipsToSet" }, "SpatialLocation": { "type": "object", "description": "The spatial location information, such as coordinates and CRS information.", "$ref": "#/definitions/osdu:wks:AbstractSpatialLocation:1.0.0" } }, "$id": "definitions/wellboreData" }, "identification": { "description": "The unique key tags associated with this entity.", "title": "unique Keys", "type": "object", "properties": { "uwi": { "description": "The unique wellbore identifier, also referred to as the API number, US well number, or UBHI. Codes can have 10, 12, or 14 digits, depending on the availability of directional sidetrack (2 digits) and event sequence codes (2 digits).", "x-slb-aliasProperties": [ "ocean:UWI", "witsml:SuffixAPI", "drillplan:uwi" ], "title": "Unique Wellbore Identifier", "type": "string", "x-osdu-natural-key": 1, "example": [ "SP435844921288", "42-501-20130-01-02" ] }, "wellbore_name": { "description": "Wellbore Name", "title": "Wellbore Name", "type": "string", "x-osdu-natural-key": 2, "example": "SMP G09995 001S0B1" }, "wellbore_number": { "description": "Wellbore Number", "title": "Wellbore Number", "x-osdu-natural-key": 3, "type": "string", "example": "001S0B1" } } }, "location": { "type": "object", "properties": { "latitude": { "description": "latitude", "type": "number", "title": "latitude", "example": [ 30 ] }, "longitude": { "description": "longitude", "type": "number", "title": "longitude", "example": [ 30 ] }, "crs": { "description": "Wellbore location CRS", "title": "CRS", "type": "string", "example": "World Geodetic System 1984" }, "epsg_code": { "description": "EPSG code of the CRS", "title": "EPSG Code", "type": "number", "example": "4326" }, "type": { "description": "Wellbore location type", "title": "Location Type", "type": "string", "example": "Surface Location" } } }, "relationshipsToSet": { "description": "The relationships tags associated with this entity.", "title": "relationships", "type": "object", "properties": { "well": { "description": "The relationship between the wellbore and this entity.", "title": "well", "type": "object", "properties": { "parent_uwi": { "description": "The unique parent well identifier.", "x-slb-aliasProperties": [ "ocean:UWI", "witsml:SuffixAPI", "drillplan:uwi" ], "title": "Unique Well Identifier", "type": "string", "example": [ "SP435844921288", "42-501-20130-01-02" ] } } } } }, "parentProjectIds": { "type": "object", "description": "Project IDs", "properties": { "projectId": { "description": "The project ID reference.", "x-osdu-relationship": [ { "EntityType": "project", "GroupType": "master-data" } ], "type": "array", "items": { "type": "string" } } } }, "parentList": { "description": "A list of entity IDs in the data ecosystem, which act as legal parents to the current entity.", "title": "Parent List", "type": "object", "properties": { "parents": { "description": "An array of none, one or many entity references in the data ecosystem, which identify the source of data in the legal sense. Example: The 'parents' are queried when, for example, the subscription of source data services is terminated. Access to the derivatives is also terminated.", "title": "Parents", "type": "array", "items": { "type": "string" }, "example": [] } } }, "metaItem": { "description": "A metadata item that allows the association of named properties or property values to a Unit/Measurement/CRS/Azimuth/Time context.", "title": "Frame of Reference Metadata Item", "type": "object", "properties": { "name": { "description": "The name of the CRS, or the symbol/name of the unit", "title": "Name or Symbol", "type": "string", "example": [ "ftUS", "NAD27 * OGP-Usa Conus / North Dakota South [32021,15851]" ] }, "propertyValues": { "description": "The list of property values, to which this metadata item provides Unit/CRS context. Typically a unit symbol is a value to a data structure. This symbol is then registered in this propertyValues array, and the persistableReference provides the absolute reference.", "title": "Attribute Names", "type": "array", "items": { "type": "string" }, "example": [ "Foot US", "ftUS" ] }, "persistableReference": { "description": "The persistable reference string that uniquely identifies the CRS or Unit.", "title": "Persistable Reference", "type": "string", "example": "{\"scaleOffset\":{\"scale\":0.3048006096012192,\"offset\":0.0},\"symbol\":\"ftUS\",\"baseMeasurement\":{\"ancestry\":\"Length\",\"type\":\"UM\"},\"type\":\"USO\"}" }, "uncertainty": { "title": "Uncertainty", "type": "number", "description": "The uncertainty of the values measured given the unit or CRS unit." }, "kind": { "description": "The kind of reference, unit, measurement, CRS, or azimuth reference.", "title": "Reference Kind", "type": "string", "example": [ "Unit", "CRS", "Measurement", "AzimuthReference" ] }, "propertyNames": { "description": "The list of property names to which this meta data item provides Unit/CRS context. Data structures that come in a single frame of reference can register the property name, but others require a full path like \"data.structureA.propertyB\" to define a unique context.", "title": "Attribute Names", "type": "array", "items": { "type": "string" }, "example": [ "elevationFromMsl", "totalDepthMdDriller", "wellHeadProjected" ] } }, "required": [ "kind", "persistableReference" ] }, "legal": { "description": "Legal metadata like legal tags, relevant other countries, and legal status.", "title": "Legal Metadata", "type": "object", "properties": { "legaltags": { "description": "The list of legal tags, see compliance API.", "title": "Legal Tags", "type": "array", "items": { "type": "string" } }, "otherRelevantDataCountries": { "description": "The list of other relevant data countries using the ISO 2-letter codes, see compliance API.", "title": "Other Relevant Data Countries", "type": "array", "items": { "type": "string" } }, "status": { "title": "Legal Status", "type": "string", "description": "The legal status." } } }, "accessControlList": { "description": "The access control tags associated with this entity.", "title": "Access Control List", "type": "object", "properties": { "viewers": { "description": "The data viewers group specification.", "title": "Owners", "type": "array", "items": { "type": "string" }, "example": [ "data.default.viewers@{domain}.com" ] }, "owners": { "description": "The data owner group specification.", "title": "Owners", "type": "array", "items": { "type": "string" }, "example": [ "data.default.owners@{domain}.com" ] } } }, "tags": { "description": "A generic dictionary of string keys mapping to string value. Only strings are permitted as keys and values.", "additionalProperties": { "type": "string" }, "title": "Tag Dictionary", "type": "object", "example": { "NameOfKey": "String value" } }, "osdu:wks:AbstractFeatureCollection:1.0.0": { "x-osdu-inheriting-from-kind": [], "x-osdu-license": "Copyright 2021, The Open Group \\nLicensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 . Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.", "$schema": "http://json-schema.org/draft-07/schema#", "x-osdu-schema-source": "osdu:wks:AbstractFeatureCollection:1.0.0", "description": "GeoJSON feature collection as originally published in https://geojson.org/schema/FeatureCollection.json. Attention: The coordinate order is fixed, Longitude first, followed by Latitude, and optionally height above MSL (EPSG:5714) as a third coordinate.", "title": "GeoJSON FeatureCollection", "type": "object", "required": [ "type", "features" ], "properties": { "type": { "type": "string", "enum": [ "FeatureCollection" ] }, "features": { "type": "array", "items": { "title": "GeoJSON Feature", "type": "object", "required": [ "type", "properties", "geometry" ], "properties": { "geometry": { "oneOf": [ { "type": "null" }, { "title": "GeoJSON Point", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "minItems": 2, "type": "array", "items": { "type": "number" } }, "type": { "type": "string", "enum": [ "Point" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "GeoJSON LineString", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "minItems": 2, "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } }, "type": { "type": "string", "enum": [ "LineString" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "GeoJSON Polygon", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "type": "array", "items": { "minItems": 4, "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } } }, "type": { "type": "string", "enum": [ "Polygon" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "GeoJSON MultiPoint", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } }, "type": { "type": "string", "enum": [ "MultiPoint" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "GeoJSON MultiLineString", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "type": "array", "items": { "minItems": 2, "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } } }, "type": { "type": "string", "enum": [ "MultiLineString" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "GeoJSON MultiPolygon", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "type": "array", "items": { "type": "array", "items": { "minItems": 4, "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } } } }, "type": { "type": "string", "enum": [ "MultiPolygon" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "GeoJSON GeometryCollection", "type": "object", "required": [ "type", "geometries" ], "properties": { "type": { "type": "string", "enum": [ "GeometryCollection" ] }, "geometries": { "type": "array", "items": { "oneOf": [ { "title": "GeoJSON Point", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "minItems": 2, "type": "array", "items": { "type": "number" } }, "type": { "type": "string", "enum": [ "Point" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "GeoJSON LineString", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "minItems": 2, "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } }, "type": { "type": "string", "enum": [ "LineString" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "GeoJSON Polygon", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "type": "array", "items": { "minItems": 4, "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } } }, "type": { "type": "string", "enum": [ "Polygon" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "GeoJSON MultiPoint", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } }, "type": { "type": "string", "enum": [ "MultiPoint" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "GeoJSON MultiLineString", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "type": "array", "items": { "minItems": 2, "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } } }, "type": { "type": "string", "enum": [ "MultiLineString" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "GeoJSON MultiPolygon", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "type": "array", "items": { "type": "array", "items": { "minItems": 4, "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } } } }, "type": { "type": "string", "enum": [ "MultiPolygon" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } } ] } }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } } ] }, "type": { "type": "string", "enum": [ "Feature" ] }, "properties": { "oneOf": [ { "type": "null" }, { "type": "object" } ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } } }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } }, "$id": "https://schema.osdu.opengroup.org/json/abstract/AbstractFeatureCollection.1.0.0.json" }, "osdu:wks:AbstractAnyCrsFeatureCollection:1.0.0": { "x-osdu-inheriting-from-kind": [], "x-osdu-license": "Copyright 2021, The Open Group \\nLicensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 . Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.", "$schema": "http://json-schema.org/draft-07/schema#", "x-osdu-schema-source": "osdu:wks:AbstractAnyCrsFeatureCollection:1.0.0", "description": "A schema like GeoJSON FeatureCollection with a non-WGS 84 CRS context, based on https://geojson.org/schema/FeatureCollection.json. Attention: The coordinate order is fixed: Longitude/Easting/Westing/X first, followed by Latitude/Northing/Southing/Y, and optionally height as a third coordinate.", "title": "AbstractAnyCrsFeatureCollection", "type": "object", "required": [ "type", "persistableReferenceCrs", "features" ], "properties": { "CoordinateReferenceSystemID": { "pattern": "^[\\w\\-\\.]+:reference-data\\-\\-CoordinateReferenceSystem:[\\w\\-\\.\\:\\%]+:[0-9]*$", "description": "The CRS reference into the CoordinateReferenceSystem catalog.", "x-osdu-relationship": [ { "EntityType": "CoordinateReferenceSystem", "GroupType": "reference-data" } ], "title": "Coordinate Reference System ID", "type": "string", "example": "namespace:reference-data--CoordinateReferenceSystem:BoundCRS.SLB.32021.15851:" }, "persistableReferenceCrs": { "description": "The CRS reference as a persistableReference string. If populated, the CoordinateReferenceSystemID takes precedence.", "type": "string", "title": "CRS Reference", "example": "{\"lateBoundCRS\":{\"wkt\":\"PROJCS[\\\"NAD_1927_StatePlane_North_Dakota_South_FIPS_3302\\\",GEOGCS[\\\"GCS_North_American_1927\\\",DATUM[\\\"D_North_American_1927\\\",SPHEROID[\\\"Clarke_1866\\\",6378206.4,294.9786982]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],PROJECTION[\\\"Lambert_Conformal_Conic\\\"],PARAMETER[\\\"False_Easting\\\",2000000.0],PARAMETER[\\\"False_Northing\\\",0.0],PARAMETER[\\\"Central_Meridian\\\",-100.5],PARAMETER[\\\"Standard_Parallel_1\\\",46.1833333333333],PARAMETER[\\\"Standard_Parallel_2\\\",47.4833333333333],PARAMETER[\\\"Latitude_Of_Origin\\\",45.6666666666667],UNIT[\\\"Foot_US\\\",0.304800609601219],AUTHORITY[\\\"EPSG\\\",32021]]\",\"ver\":\"PE_10_3_1\",\"name\":\"NAD_1927_StatePlane_North_Dakota_South_FIPS_3302\",\"authCode\":{\"auth\":\"EPSG\",\"code\":\"32021\"},\"type\":\"LBC\"},\"singleCT\":{\"wkt\":\"GEOGTRAN[\\\"NAD_1927_To_WGS_1984_79_CONUS\\\",GEOGCS[\\\"GCS_North_American_1927\\\",DATUM[\\\"D_North_American_1927\\\",SPHEROID[\\\"Clarke_1866\\\",6378206.4,294.9786982]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],GEOGCS[\\\"GCS_WGS_1984\\\",DATUM[\\\"D_WGS_1984\\\",SPHEROID[\\\"WGS_1984\\\",6378137.0,298.257223563]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],METHOD[\\\"NADCON\\\"],PARAMETER[\\\"Dataset_conus\\\",0.0],AUTHORITY[\\\"EPSG\\\",15851]]\",\"ver\":\"PE_10_3_1\",\"name\":\"NAD_1927_To_WGS_1984_79_CONUS\",\"authCode\":{\"auth\":\"EPSG\",\"code\":\"15851\"},\"type\":\"ST\"},\"ver\":\"PE_10_3_1\",\"name\":\"NAD27 * OGP-Usa Conus / North Dakota South [32021,15851]\",\"authCode\":{\"auth\":\"SLB\",\"code\":\"32021079\"},\"type\":\"EBC\"}" }, "features": { "type": "array", "items": { "title": "AnyCrsGeoJSON Feature", "type": "object", "required": [ "type", "properties", "geometry" ], "properties": { "geometry": { "oneOf": [ { "type": "null" }, { "title": "AnyCrsGeoJSON Point", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "minItems": 2, "type": "array", "items": { "type": "number" } }, "type": { "type": "string", "enum": [ "AnyCrsPoint" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "AnyCrsGeoJSON LineString", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "minItems": 2, "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } }, "type": { "type": "string", "enum": [ "AnyCrsLineString" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "AnyCrsGeoJSON Polygon", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "type": "array", "items": { "minItems": 4, "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } } }, "type": { "type": "string", "enum": [ "AnyCrsPolygon" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "AnyCrsGeoJSON MultiPoint", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } }, "type": { "type": "string", "enum": [ "AnyCrsMultiPoint" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "AnyCrsGeoJSON MultiLineString", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "type": "array", "items": { "minItems": 2, "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } } }, "type": { "type": "string", "enum": [ "AnyCrsMultiLineString" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "AnyCrsGeoJSON MultiPolygon", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "type": "array", "items": { "type": "array", "items": { "minItems": 4, "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } } } }, "type": { "type": "string", "enum": [ "AnyCrsMultiPolygon" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "AnyCrsGeoJSON GeometryCollection", "type": "object", "required": [ "type", "geometries" ], "properties": { "type": { "type": "string", "enum": [ "AnyCrsGeometryCollection" ] }, "geometries": { "type": "array", "items": { "oneOf": [ { "title": "AnyCrsGeoJSON Point", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "minItems": 2, "type": "array", "items": { "type": "number" } }, "type": { "type": "string", "enum": [ "AnyCrsPoint" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "AnyCrsGeoJSON LineString", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "minItems": 2, "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } }, "type": { "type": "string", "enum": [ "AnyCrsLineString" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "AnyCrsGeoJSON Polygon", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "type": "array", "items": { "minItems": 4, "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } } }, "type": { "type": "string", "enum": [ "AnyCrsPolygon" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "AnyCrsGeoJSON MultiPoint", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } }, "type": { "type": "string", "enum": [ "AnyCrsMultiPoint" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "AnyCrsGeoJSON MultiLineString", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "type": "array", "items": { "minItems": 2, "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } } }, "type": { "type": "string", "enum": [ "AnyCrsMultiLineString" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "AnyCrsGeoJSON MultiPolygon", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "type": "array", "items": { "type": "array", "items": { "minItems": 4, "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } } } }, "type": { "type": "string", "enum": [ "AnyCrsMultiPolygon" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } } ] } }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } } ] }, "type": { "type": "string", "enum": [ "AnyCrsFeature" ] }, "properties": { "oneOf": [ { "type": "null" }, { "type": "object" } ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } } }, "persistableReferenceUnitZ": { "description": "The unit of measure for the Z-axis (only for 3-dimensional coordinates, where the CRS does not describe the vertical unit). Note that the direction is upwards positive, therefore Z means height.", "type": "string", "title": "Z-Unit Reference", "example": "{\"scaleOffset\":{\"scale\":1.0,\"offset\":0.0},\"symbol\":\"m\",\"baseMeasurement\":{\"ancestry\":\"Length\",\"type\":\"UM\"},\"type\":\"USO\"}" }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } }, "persistableReferenceVerticalCrs": { "description": "The VerticalCRS reference as persistableReference string. If populated, the VerticalCoordinateReferenceSystemID takes precedence. The property is null or empty for 2D geometries. For 3D geometries and absent or null persistableReferenceVerticalCrs, the vertical CRS is either provided using the persistableReferenceCrs's CompoundCRS or it is implicitly defined as EPSG:5714 MSL height.", "type": "string", "title": "Vertical CRS Reference", "example": "{\"authCode\":{\"auth\":\"EPSG\",\"code\":\"5773\"},\"type\":\"LBC\",\"ver\":\"PE_10_3_1\",\"name\":\"EGM96_Geoid\",\"wkt\":\"VERTCS[\\\"EGM96_Geoid\\\",VDATUM[\\\"EGM96_Geoid\\\"],PARAMETER[\\\"Vertical_Shift\\\",0.0],PARAMETER[\\\"Direction\\\",1.0],UNIT[\\\"Meter\\\",1.0],AUTHORITY[\\\"EPSG\\\",5773]]\"}" }, "type": { "type": "string", "enum": [ "AnyCrsFeatureCollection" ] }, "VerticalCoordinateReferenceSystemID": { "pattern": "^[\\w\\-\\.]+:reference-data\\-\\-CoordinateReferenceSystem:[\\w\\-\\.\\:\\%]+:[0-9]*$", "description": "The explicit VerticalCRS reference into the CoordinateReferenceSystem catalog. This property stays empty for 2D geometries. Absent or empty values for 3D geometries mean the context may be provided by a CompoundCRS in the 'CoordinateReferenceSystemID' or implicitly as EPSG:5714 MSL height.", "x-osdu-relationship": [ { "EntityType": "CoordinateReferenceSystem", "GroupType": "reference-data" } ], "title": "Vertical Coordinate Reference System ID", "type": "string", "example": "namespace:reference-data--CoordinateReferenceSystem:VerticalCRS.EPSG.5773:" } }, "$id": "https://schema.osdu.opengroup.org/json/abstract/AbstractAnyCrsFeatureCollection.1.0.0.json" }, "osdu:wks:AbstractFeatureCollection:1.0.0": { "x-osdu-inheriting-from-kind": [], "x-osdu-license": "Copyright 2021, The Open Group \\nLicensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 . Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.", "$schema": "http://json-schema.org/draft-07/schema#", "x-osdu-schema-source": "osdu:wks:AbstractFeatureCollection:1.0.0", "description": "GeoJSON feature collection as originally published in https://geojson.org/schema/FeatureCollection.json. Attention: The coordinate order is fixed, Longitude first, followed by Latitude, and optionally height above MSL (EPSG:5714) as a third coordinate.", "title": "GeoJSON FeatureCollection", "type": "object", "required": [ "type", "features" ], "properties": { "type": { "type": "string", "enum": [ "FeatureCollection" ] }, "features": { "type": "array", "items": { "title": "GeoJSON Feature", "type": "object", "required": [ "type", "properties", "geometry" ], "properties": { "geometry": { "oneOf": [ { "type": "null" }, { "title": "GeoJSON Point", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "minItems": 2, "type": "array", "items": { "type": "number" } }, "type": { "type": "string", "enum": [ "Point" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "GeoJSON LineString", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "minItems": 2, "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } }, "type": { "type": "string", "enum": [ "LineString" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "GeoJSON Polygon", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "type": "array", "items": { "minItems": 4, "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } } }, "type": { "type": "string", "enum": [ "Polygon" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "GeoJSON MultiPoint", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } }, "type": { "type": "string", "enum": [ "MultiPoint" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "GeoJSON MultiLineString", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "type": "array", "items": { "minItems": 2, "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } } }, "type": { "type": "string", "enum": [ "MultiLineString" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "GeoJSON MultiPolygon", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "type": "array", "items": { "type": "array", "items": { "minItems": 4, "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } } } }, "type": { "type": "string", "enum": [ "MultiPolygon" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "GeoJSON GeometryCollection", "type": "object", "required": [ "type", "geometries" ], "properties": { "type": { "type": "string", "enum": [ "GeometryCollection" ] }, "geometries": { "type": "array", "items": { "oneOf": [ { "title": "GeoJSON Point", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "minItems": 2, "type": "array", "items": { "type": "number" } }, "type": { "type": "string", "enum": [ "Point" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "GeoJSON LineString", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "minItems": 2, "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } }, "type": { "type": "string", "enum": [ "LineString" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "GeoJSON Polygon", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "type": "array", "items": { "minItems": 4, "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } } }, "type": { "type": "string", "enum": [ "Polygon" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "GeoJSON MultiPoint", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } }, "type": { "type": "string", "enum": [ "MultiPoint" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "GeoJSON MultiLineString", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "type": "array", "items": { "minItems": 2, "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } } }, "type": { "type": "string", "enum": [ "MultiLineString" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "GeoJSON MultiPolygon", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "type": "array", "items": { "type": "array", "items": { "minItems": 4, "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } } } }, "type": { "type": "string", "enum": [ "MultiPolygon" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } } ] } }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } } ] }, "type": { "type": "string", "enum": [ "Feature" ] }, "properties": { "oneOf": [ { "type": "null" }, { "type": "object" } ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } } }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } }, "$id": "https://schema.osdu.opengroup.org/json/abstract/AbstractFeatureCollection.1.0.0.json" }, "osdu:wks:AbstractAnyCrsFeatureCollection:1.0.0": { "x-osdu-inheriting-from-kind": [], "x-osdu-license": "Copyright 2021, The Open Group \\nLicensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 . Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.", "$schema": "http://json-schema.org/draft-07/schema#", "x-osdu-schema-source": "osdu:wks:AbstractAnyCrsFeatureCollection:1.0.0", "description": "A schema like GeoJSON FeatureCollection with a non-WGS 84 CRS context, based on https://geojson.org/schema/FeatureCollection.json. Attention: The coordinate order is fixed, Longitude/Easting/Westing/X first, followed by Latitude/Northing/Southing/Y, and optionally height as a third coordinate.", "title": "AbstractAnyCrsFeatureCollection", "type": "object", "required": [ "type", "persistableReferenceCrs", "features" ], "properties": { "CoordinateReferenceSystemID": { "pattern": "^[\\w\\-\\.]+:reference-data\\-\\-CoordinateReferenceSystem:[\\w\\-\\.\\:\\%]+:[0-9]*$", "description": "The CRS reference into the CoordinateReferenceSystem catalog.", "x-osdu-relationship": [ { "EntityType": "CoordinateReferenceSystem", "GroupType": "reference-data" } ], "title": "Coordinate Reference System ID", "type": "string", "example": "namespace:reference-data--CoordinateReferenceSystem:BoundCRS.SLB.32021.15851:" }, "persistableReferenceCrs": { "description": "The CRS reference as a persistableReference string. If populated, the CoordinateReferenceSystemID takes precedence.", "type": "string", "title": "CRS Reference", "example": "{\"lateBoundCRS\":{\"wkt\":\"PROJCS[\\\"NAD_1927_StatePlane_North_Dakota_South_FIPS_3302\\\",GEOGCS[\\\"GCS_North_American_1927\\\",DATUM[\\\"D_North_American_1927\\\",SPHEROID[\\\"Clarke_1866\\\",6378206.4,294.9786982]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],PROJECTION[\\\"Lambert_Conformal_Conic\\\"],PARAMETER[\\\"False_Easting\\\",2000000.0],PARAMETER[\\\"False_Northing\\\",0.0],PARAMETER[\\\"Central_Meridian\\\",-100.5],PARAMETER[\\\"Standard_Parallel_1\\\",46.1833333333333],PARAMETER[\\\"Standard_Parallel_2\\\",47.4833333333333],PARAMETER[\\\"Latitude_Of_Origin\\\",45.6666666666667],UNIT[\\\"Foot_US\\\",0.304800609601219],AUTHORITY[\\\"EPSG\\\",32021]]\",\"ver\":\"PE_10_3_1\",\"name\":\"NAD_1927_StatePlane_North_Dakota_South_FIPS_3302\",\"authCode\":{\"auth\":\"EPSG\",\"code\":\"32021\"},\"type\":\"LBC\"},\"singleCT\":{\"wkt\":\"GEOGTRAN[\\\"NAD_1927_To_WGS_1984_79_CONUS\\\",GEOGCS[\\\"GCS_North_American_1927\\\",DATUM[\\\"D_North_American_1927\\\",SPHEROID[\\\"Clarke_1866\\\",6378206.4,294.9786982]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],GEOGCS[\\\"GCS_WGS_1984\\\",DATUM[\\\"D_WGS_1984\\\",SPHEROID[\\\"WGS_1984\\\",6378137.0,298.257223563]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],METHOD[\\\"NADCON\\\"],PARAMETER[\\\"Dataset_conus\\\",0.0],AUTHORITY[\\\"EPSG\\\",15851]]\",\"ver\":\"PE_10_3_1\",\"name\":\"NAD_1927_To_WGS_1984_79_CONUS\",\"authCode\":{\"auth\":\"EPSG\",\"code\":\"15851\"},\"type\":\"ST\"},\"ver\":\"PE_10_3_1\",\"name\":\"NAD27 * OGP-Usa Conus / North Dakota South [32021,15851]\",\"authCode\":{\"auth\":\"SLB\",\"code\":\"32021079\"},\"type\":\"EBC\"}" }, "features": { "type": "array", "items": { "title": "AnyCrsGeoJSON Feature", "type": "object", "required": [ "type", "properties", "geometry" ], "properties": { "geometry": { "oneOf": [ { "type": "null" }, { "title": "AnyCrsGeoJSON Point", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "minItems": 2, "type": "array", "items": { "type": "number" } }, "type": { "type": "string", "enum": [ "AnyCrsPoint" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "AnyCrsGeoJSON LineString", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "minItems": 2, "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } }, "type": { "type": "string", "enum": [ "AnyCrsLineString" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "AnyCrsGeoJSON Polygon", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "type": "array", "items": { "minItems": 4, "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } } }, "type": { "type": "string", "enum": [ "AnyCrsPolygon" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "AnyCrsGeoJSON MultiPoint", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } }, "type": { "type": "string", "enum": [ "AnyCrsMultiPoint" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "AnyCrsGeoJSON MultiLineString", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "type": "array", "items": { "minItems": 2, "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } } }, "type": { "type": "string", "enum": [ "AnyCrsMultiLineString" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "AnyCrsGeoJSON MultiPolygon", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "type": "array", "items": { "type": "array", "items": { "minItems": 4, "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } } } }, "type": { "type": "string", "enum": [ "AnyCrsMultiPolygon" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "AnyCrsGeoJSON GeometryCollection", "type": "object", "required": [ "type", "geometries" ], "properties": { "type": { "type": "string", "enum": [ "AnyCrsGeometryCollection" ] }, "geometries": { "type": "array", "items": { "oneOf": [ { "title": "AnyCrsGeoJSON Point", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "minItems": 2, "type": "array", "items": { "type": "number" } }, "type": { "type": "string", "enum": [ "AnyCrsPoint" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "AnyCrsGeoJSON LineString", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "minItems": 2, "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } }, "type": { "type": "string", "enum": [ "AnyCrsLineString" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "AnyCrsGeoJSON Polygon", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "type": "array", "items": { "minItems": 4, "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } } }, "type": { "type": "string", "enum": [ "AnyCrsPolygon" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "AnyCrsGeoJSON MultiPoint", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } }, "type": { "type": "string", "enum": [ "AnyCrsMultiPoint" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "AnyCrsGeoJSON MultiLineString", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "type": "array", "items": { "minItems": 2, "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } } }, "type": { "type": "string", "enum": [ "AnyCrsMultiLineString" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } }, { "title": "AnyCrsGeoJSON MultiPolygon", "type": "object", "required": [ "type", "coordinates" ], "properties": { "coordinates": { "type": "array", "items": { "type": "array", "items": { "minItems": 4, "type": "array", "items": { "minItems": 2, "type": "array", "items": { "type": "number" } } } } }, "type": { "type": "string", "enum": [ "AnyCrsMultiPolygon" ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } } ] } }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } } ] }, "type": { "type": "string", "enum": [ "AnyCrsFeature" ] }, "properties": { "oneOf": [ { "type": "null" }, { "type": "object" } ] }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } } } } }, "persistableReferenceUnitZ": { "description": "The unit of measure for the Z-axis (only for 3-dimensional coordinates, where the CRS does not describe the vertical unit). Note that the direction is upwards positive, therefore Z means height.", "type": "string", "title": "Z-Unit Reference", "example": "{\"scaleOffset\":{\"scale\":1.0,\"offset\":0.0},\"symbol\":\"m\",\"baseMeasurement\":{\"ancestry\":\"Length\",\"type\":\"UM\"},\"type\":\"USO\"}" }, "bbox": { "minItems": 4, "type": "array", "items": { "type": "number" } }, "persistableReferenceVerticalCrs": { "description": "The VerticalCRS reference as a persistableReference string. If populated, the VerticalCoordinateReferenceSystemID takes precedence. The property is null or empty for 2D geometries. For 3D geometries and absent or null persistableReferenceVerticalCrs, the vertical CRS is either provided using the persistableReferenceCrs's CompoundCRS or it is implicitly defined as EPSG:5714 MSL height.", "type": "string", "title": "Vertical CRS Reference", "example": "{\"authCode\":{\"auth\":\"EPSG\",\"code\":\"5773\"},\"type\":\"LBC\",\"ver\":\"PE_10_3_1\",\"name\":\"EGM96_Geoid\",\"wkt\":\"VERTCS[\\\"EGM96_Geoid\\\",VDATUM[\\\"EGM96_Geoid\\\"],PARAMETER[\\\"Vertical_Shift\\\",0.0],PARAMETER[\\\"Direction\\\",1.0],UNIT[\\\"Meter\\\",1.0],AUTHORITY[\\\"EPSG\\\",5773]]\"}" }, "type": { "type": "string", "enum": [ "AnyCrsFeatureCollection" ] }, "VerticalCoordinateReferenceSystemID": { "pattern": "^[\\w\\-\\.]+:reference-data\\-\\-CoordinateReferenceSystem:[\\w\\-\\.\\:\\%]+:[0-9]*$", "description": "The explicit VerticalCRS reference into the CoordinateReferenceSystem catalog. This property stays empty for 2D geometries. Absent or empty values for 3D geometries mean the context may be provided by a CompoundCRS in the 'CoordinateReferenceSystemID' or implicitly with EPSG:5714 MSL height.", "x-osdu-relationship": [ { "EntityType": "CoordinateReferenceSystem", "GroupType": "reference-data" } ], "title": "Vertical Coordinate Reference System ID", "type": "string", "example": "namespace:reference-data--CoordinateReferenceSystem:VerticalCRS.EPSG.5773:" } }, "$id": "https://schema.osdu.opengroup.org/json/abstract/AbstractAnyCrsFeatureCollection.1.0.0.json" }, "osdu:wks:AbstractSpatialLocation:1.0.0": { "x-osdu-inheriting-from-kind": [], "x-osdu-license": "Copyright 2021, The Open Group \\nLicensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 . Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.", "$schema": "http://json-schema.org/draft-07/schema#", "x-osdu-schema-source": "osdu:wks:AbstractSpatialLocation:1.0.0", "description": "A geographic object that can be described by a set of points.", "title": "AbstractSpatialLocation", "type": "object", "properties": { "AsIngestedCoordinates": { "description": "The original or 'as ingested' coordinates (Point, MultiPoint, LineString, MultiLineString, Polygon, or MultiPolygon). The name 'AsIngestedCoordinates' was chosen to contrast it with 'OriginalCoordinates', which carries the uncertainty whether any coordinate operations took place before ingestion. In cases where the original CRS is different from the as-ingested CRS, the OperationsApplied can also contain the list of operations applied to the coordinate prior to ingestion. The data structure is similar to the GeoJSON FeatureCollection; however in a CRS context it is explicitly defined within the AbstractAnyCrsFeatureCollection. The coordinate sequence follows the GeoJSON standard, such as 'eastward/longitude', 'northward/latitude', 'upward/height' unless overridden by an explicit direction in the AsIngestedCoordinates.VerticalCoordinateReferenceSystemID.", "x-osdu-frame-of-reference": "CRS:", "title": "As Ingested Coordinates", "$ref": "#/definitions/osdu:wks:AbstractAnyCrsFeatureCollection:1.0.0" }, "SpatialParameterTypeID": { "x-osdu-relationship": [ { "EntityType": "SpatialParameterType", "GroupType": "reference-data" } ], "pattern": "^[\\w\\-\\.]+:reference-data\\-\\-SpatialParameterType:[\\w\\-\\.\\:\\%]+:[0-9]*$", "description": "A type of spatial representation of an object, often general, such as an Outline, which could be applied to Field, Reservoir, Facility, etc., or sometimes specific, such as Onshore Outline, State Offshore Outline, or Federal Offshore Outline, 3 spatial representations that may be used by Countries).", "type": "string" }, "QuantitativeAccuracyBandID": { "x-osdu-relationship": [ { "EntityType": "QuantitativeAccuracyBand", "GroupType": "reference-data" } ], "pattern": "^[\\w\\-\\.]+:reference-data\\-\\-QuantitativeAccuracyBand:[\\w\\-\\.\\:\\%]+:[0-9]*$", "description": "An approximate quantitative assessment of the quality of a location (accurate to > 500 m (not very accurate)), to < 1 m, etc.", "type": "string" }, "CoordinateQualityCheckRemarks": { "description": "Freetext remarks on Quality Check.", "type": "array", "items": { "type": "string" } }, "AppliedOperations": { "description": "The audit trail of operations applied to the coordinates from the original state to the current state. The list may contain operations applied prior to ingestion as well as the operations applied to produce the Wgs84Coordinates. The text elements refer to ESRI style CRS and Transformation names, which may have to be translated to EPSG standard names.", "title": "Operations Applied", "type": "array", "items": { "type": "string" }, "example": [ "conversion from ED_1950_UTM_Zone_31N to GCS_European_1950; 1 points converted", "transformation GCS_European_1950 to GCS_WGS_1984 using ED_1950_To_WGS_1984_24; 1 points successfully transformed" ] }, "QualitativeSpatialAccuracyTypeID": { "x-osdu-relationship": [ { "EntityType": "QualitativeSpatialAccuracyType", "GroupType": "reference-data" } ], "pattern": "^[\\w\\-\\.]+:reference-data\\-\\-QualitativeSpatialAccuracyType:[\\w\\-\\.\\:\\%]+:[0-9]*$", "description": "A qualitative description of the quality of a spatial location, for example, unverifiable, not verified, or basic validation.", "type": "string" }, "CoordinateQualityCheckPerformedBy": { "description": "The user who performed the Quality Check.", "type": "string" }, "SpatialLocationCoordinatesDate": { "format": "date-time", "description": "Date when coordinates were measured or retrieved.", "x-osdu-frame-of-reference": "DateTime", "type": "string" }, "CoordinateQualityCheckDateTime": { "format": "date-time", "description": "The date of the Quality Check.", "x-osdu-frame-of-reference": "DateTime", "type": "string" }, "Wgs84Coordinates": { "description": "The normalized coordinates (Point, MultiPoint, LineString, MultiLineString, Polygon, or MultiPolygon) based on WGS 84 (EPSG:4326 for 2-dimensional coordinates, EPSG:4326 + EPSG:5714 (MSL) for 3-dimensional coordinates). This derived coordinate representation is intended for global discoverability only. The schema of this substructure is identical to the GeoJSON FeatureCollection https://geojson.org/schema/FeatureCollection.json. The coordinate sequence follows the GeoJSON standard, such as longitude, latitude, or height.", "title": "WGS 84 Coordinates", "$ref": "#/definitions/osdu:wks:AbstractFeatureCollection:1.0.0" }, "SpatialGeometryTypeID": { "x-osdu-relationship": [ { "EntityType": "SpatialGeometryType", "GroupType": "reference-data" } ], "pattern": "^[\\w\\-\\.]+:reference-data\\-\\-SpatialGeometryType:[\\w\\-\\.\\:\\%]+:[0-9]*$", "description": "Indicates the expected look of the SpatialParameterType, for example Point, MultiPoint, LineString, MultiLineString, Polygon, or MultiPolygon. The value constrains the type of geometries in the GeoJSON Wgs84Coordinates and AsIngestedCoordinates.", "type": "string" } }, "$id": "https://schema.osdu.opengroup.org/json/abstract/AbstractSpatialLocation.1.0.0.json" } }, "properties": { "data": { "description": "Wellbore data container", "title": "Wellbore Data", "$ref": "#/definitions/wellboreData" }, "ancestry": { "description": "The links to data, which constitute the inputs.", "title": "Ancestry", "$ref": "#/definitions/parentList" }, "kind": { "default": "<authority>:<source>:wellbore:1.0.0", "description": "OSDU demo wellbore kind specification", "title": "Wellbore Kind", "type": "string" }, "meta": { "description": "The metadata section that links the 'unitKey', 'crsKey' to self-contained definitions (persistableReference).", "title": "Frame of Reference Meta Data", "type": "array", "items": { "$ref": "#/definitions/metaItem" } }, "legal": { "description": "The geological interpretation's legal tags.", "title": "Legal Tags", "$ref": "#/definitions/legal" }, "acl": { "description": "The access control tags associated with this entity.", "title": "Access Control List", "$ref": "#/definitions/accessControlList" }, "id": { "description": "The unique identifier of the wellbore.", "title": "Wellbore ID", "type": "string" }, "type": { "description": "The reference entity type as declared in common:metadata:entity:*.", "title": "Entity Type", "type": "string" }, "version": { "format": "int64", "description": "The version number of this wellbore; set by the framework.", "title": "Entity Version Number", "type": "number", "example": "1040815391631285" }, "tags": { "description": "The links to data, which constitute the inputs.", "title": "tags", "type": "array", "items": { "$ref": "#/definitions/tags" } } } } }
The file metadata is used to provide the ACL, Legal Tags, file source, and Target kind of the ingested data, Attributes and their kinds, Metadata information like Unit, CRS, and establish relationships with existing parents.
Sample Metadata:
Sample file: sample_metadata_wellbore.json
{
"data": {
"SchemaFormatTypeID": "string",
"Name": "Wellbore-Sample_data_v2.csv",
"EncodingFormatTypeID": "string",
"Endian": "BIG",
"Checksum": "string",
"DatasetProperties": {
"FileSourceInfo": {
"Name": "File",
"PreLoadFilePath": "string",
"FileSource": "/osdu-user/1626679045186-2021-07-19-07-17-25-186/b0e44c076fbf4ce0998a6af06891aed2",
"PreloadFileCreateUser": "string",
"PreloadFileCreateDate": "string",
"PreloadFileModifyUser": "string",
"PreloadFileModifyDate": "string",
"FileSize": "string",
"EncodingFormatTypeID": "string"
}
},
"ExtensionProperties": {
"Classification": "Raw File",
"Description": "An text further describing this file example.",
"ExternalIds": [
"string"
],
"FileDateCreated": {},
"FileDateModified": {},
"FileContentsDetails": {
"TargetKind": "<authority>:<source>:wellbore:1.0.0",
"nestedFieldDelimiter":".",
"FileType": "csv",
"SpatialMapping": {
"type": "point",
"latitude": "location.[0].latitude",
"longitude": "location.[0].longitude"
},
"FrameOfReference": [
{
"kind": "CRS",
"name": "GCS_WGS_1984",
"persistableReference": "{\"wkt\":\"GEOGCS[\\\"GCS_WGS_1984\\\",DATUM[\\\"D_WGS_1984\\\",SPHEROID[\\\"WGS_1984\\\",6378137.0,298.257223563]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433],AUTHORITY[\\\"EPSG\\\",4326]]\",\"ver\":\"PE_10_3_1\",\"name\":\"GCS_WGS_1984\",\"authCode\":{\"auth\":\"EPSG\",\"code\":\"4326\"},\"type\":\"LBC\"}",
"propertyNames": [
"location.[0].latitude",
"location.[0].longitude",
"location.[1].latitude",
"location.[1].longitude"
],
"propertyValues": [
"deg"
],
"uncertainty": 0
},
{
"kind": "DateTime",
"persistableReference": "{\"type\": \"DAT\", \"format\": \"MM-dd-yyyy\"}",
"name": "date",
"propertyNames": [
"permit.[1]",
"initial_completion_date",
"status_date",
"spud_date"
],
"propertyValues": [
"permit.[1]",
"initial_completion_date",
"status_date",
"Sspud_date"
],
"uncertainty": 0
},
{
"kind": "Unit",
"name": "ft",
"persistableReference": "{\"scaleOffset\":{\"scale\":0.3048,\"offset\":0.0},\"symbol\":\"ft\",\"baseMeasurement\":{\"ancestry\":\"Length\",\"type\":\"UM\"},\"type\":\"USO\"}",
"propertyNames": [
"md",
"tvd",
"elevation"
],
"propertyValues": [
"ft"
],
"uncertainty": 0
}
],
"relationships": [
{
"project": {
"ids": [
"<authority>:<source>:project-sxs0f1a5219-4640-50af-9f63:",
"<authority>:<source>:project-dsv05f45665-5885-40ad-d9m3:"
]
}
}
],
"relatedNaturalKey": [
{
"well": {
"targetKind": "<authority>:<source>:well:1.0.0",
"keys":[
{
"sourceColumn": "relationshipsToSet.parent_uwi",
"targetAttribute": "UWI"
}
]
}
},
{
"states": {
"targetKind": "<authority>:<source>:states:1.0.0",
"keys":[
{
"sourceColumn": "state",
"targetAttribute": "STATE_NAME"
}
]
}
},
{
"trajectory": {
"keys":[
{
"sourceColumn": "parentDefTrajectory"
}
]
}
},
{
"field": {
"keys":[
{
"sourceColumn": "parentField"
}
]
}
}
]
}
}
},
"meta": [],
"id": "<authority>:dataset--File.Generic:dec44cb4-916c-4bee-85da-0de713a32f26",
"version": 1610634966669814,
"kind": "<authority>:wks:dataset--File.Generic:1.0.0",
"acl": {
"viewers": [
"data.default.viewers@{domain}.com"
],
"owners": [
"data.default.viewers@{domain}.com"
]
},
"legal": {
"legaltags": [
"<valid legal tag>"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
}
}ACL:
Specifies the Access Control list
Example:
"acl": { "viewers": [ "data.default.viewers@{domain}.com" ], "owners": [ "data.default.viewers@{domain}.com" ] }
Legal Tags:
- Specifies the legal tags. This holds the details of the data compliance for the data ingested.
- API Reference
- Example:
"legal": { "legaltags": [ "opendes-public-usa-dataset-7643990" ], "otherRelevantDataCountries": [ "US" ], "status": "compliant" }
File source:
- This is the relative file path of the CSV file uploaded in the file-staging-area.
- This value is the relative file path that you received from the
Get Signed Urlresponse as (mentioned above in Step #2).
Target kind / Schema kind:
- This is the Schema kind in which data is ingested in the following format
<<authority>:<source>:<entityType>:<version>>.
- This is the Schema kind in which data is ingested in the following format
Nested / Nested array schema
To support the ingestion of data into nested/nested array attributes, the headers of the uploaded CSV header must match the nested/nested array attributes of the target schemas using the delimiter characters defined on the metadata file.
The
nestedFieldDelimiterattribute in the file metadata is used to define which character to use in the CSV file header to describe the different levels of nested/nested array attributes while the ingestor parses the files.The delimiter character used to define nested/nested array structures on the CSV file header must match the one defined by the
nestedFieldDelimiterin the file metadata record, otherwise the attributes on the CSV file will not be considered nested/nested array.{ "ExtensionProperties": { "FileContentsDetails": { "TargetKind": "<<authority>:<source>:<entityType>:<version>>", "nestedFieldDelimiter":".", "FileType": "csv" } } }
Unit
- This value is used for converting the declared frame of reference information into the appropriate persistable reference as per the OSDU Data Platform Unit service. This information is stored in the meta[] block.
- The ExtensionProperties block is used to provide content details of the file. The CSV ingestion workflow uses this same block for the Unit information.
CRS
- The CRS Frame of Reference information must be included in the schema of the data, including the source CRS (either geographic or projected), and in the case of projected. This CRS info and, if provided in schema, persistable reference information is stored in the meta[] block.
- The ExtensionProperties block is used to provide the content details of the file. The CSV ingestion workflow uses this same block for the CRS information.
Relationships
- CSV ingestion supports two kinds of relationships:
- Deterministic (Schema-driven)
These relationships require that the entity be referred to in the record's targetKind schema under an attribute havingx-osdu-relationshiptag. Because they are present in the schema, they are represented directly as attributes in thedatablock of the record.- String and array type relationships are supported.
- Pattern matching is also done if the pattern is present for the matching entity in the schema.
- Non Deterministic (Data-driven)
These relationships do not require any mention in the schema. They are represented within thedata.relationshipsblock of the record. - Deterministic (Data-driven)
These relationships are established against thecopyToattribute mentioned in therelated-natural-keyblock in the File Metadata.
- Deterministic (Schema-driven)
- The ExtensionProperties block in the file metadata record is used to provide additional information for ingestion. We can use this block to provide relationship information. There are three ways of providing this information:
In the
relationshipsblock with the entity name and a list of parent record IDs. The IDs provided here are directly used to establish relationships.In the
relatedNaturalKeyblock as an entity that requires a search of the targetKind using one or more attributes as the natural key to establish a relationship.- sourceColumn: Column name of the CSV file which refers to the key parent attribute.
- targetKind: Schema ID of the parent record.
- targetAttribute: The key attribute of the parent record which is used to search the parent record. If the
targetAttributeis a nested attribute, it must be.separated.
Pre-requisites: The CSV file must have the key attributes of the parent records.
In the
relatedNaturalKeyblock as an entity that has the related parent record ID directly in the CSV file under the sourceColumn of the keys block.- sourceColumn: The column name of the CSV file which refers to the parent record ID.
Pre-requisites: The parent record ID must be present under the sourceColumn of the CSV file.
{ "ExtensionProperties": { "FileContentsDetails": { "TargetKind": "<authority>:<source>:wellbore:1.0.0", "nestedFieldDelimiter": ".", "FileType": "csv", "relationships": [ { "project": { "ids": [ "<authority>:<source>:project-sxs0f1a5219-4640-50af-9f63:", "<authority>:<source>:project-dsv05f45665-5885-40ad-d9m3:" ] }, "states": { "ids": [ "<authority>:<source>:project-sxs0f1a5219-4640-50af-3r34:", "<authority>:<source>:project-dsv05f45665-5885-40ad-3m63:" ] } } ], "relatedNaturalKey": [ { "well": { "targetKind": "<authority>:<source>:well:1.0.0", "keys": [ { "sourceColumn": "relationshipsToSet.parent_uwi", "targetAttribute": "UWI" } ] } }, { "Wellbore": { "targetKind": "<authority>:<source>:Wellbore:1.0.0", "copyTo": "WellboreID", "keys": [ { "sourceColumn": "relationshipsToSet.parent_uwi", "targetAttribute": "UWI" } ] } }, "states": { "targetKind": "<authority>:<source>:states:1.0.0", "keys": [ { "sourceColumn": "state_name", "targetAttribute": "STATE_NAME" }, { "sourceColumn": "state_code", "targetAttribute": "STATE_CODE" } ] } }, { "trajectory": { "keys": [ { "sourceColumn": "parentDefTrajectory" } ] } }, { "field": { "keys": [ { "sourceColumn": "parentField" } ] } } ] } } }- The schema of the record should have information about attributes that contain deterministic relationships.
- The EntityType field within the
x-osdu-relationshipblock should contain the entity that needs to be matched from the ExtensionProperties block.
- The EntityType field within the
{ "properties": { "wellId": { "x-osdu-relationship": [ { "EntityType": "well", "GroupType": "master-data" } ], "type": "string" }, "WellboreID": { "x-osdu-relationship": [ { "EntityType": "Wellbore", "GroupType": "master-data" } ], "type": "string" }, "VerticalReferenceEntityId": { "x-osdu-relationship": [ { "EntityType": "Wellbore", "GroupType": "master-data" } ], "type": "string" }, "defTrajectoryId": { "x-osdu-relationship": [ { "EntityType": "trajectory" } ], "pattern": "^[\\w\\-\\.]+:[\\w\\-\\.]+:[\\w\\-\\.]+:[\\w\\-\\.]*$", "type": "string" }, "fieldId": { "x-osdu-relationship": [ { "EntityType": "field" } ], "pattern": "^[\\w\\-\\.]+:master-data\\-\\-field:[\\w\\-\\.\\:\\%]+:[0-9]*$", "type": "string" }, "projectIds": { "description": "The relationships tags associated with this entity.", "title": "Project IDs", "type": "object", "x-osdu-indexing": { "type": "nested" }, "$ref": "#/definitions/parentProjectIds" }, "parentProjectIds": { "type": "object", "description": "Project IDs", "properties": { "projectId": { "description": "The project ID reference.", "x-osdu-relationship": [ { "EntityType": "project", "GroupType": "master-data" } ], "type": "array", "items": { "type": "string" } } } } } }- The final record will then have the relationships defined as shown below:
{ "data": { "relationships": { "field": { "ids": [ "<authority>:<source>:field-sxs0f1a5219-4640-50af-9f63-c09140d57c12:" ] }, "states": { "ids": [ "slb-osdu-prod-des-prod-testing:states:external-dGV4YXM" ] } }, "projectIds": { "parentProjectIds": { "projectId": [ "<authority>:<source>:project-sxs0f1a5219-4640-50af-9f63:", "<authority>:<source>:project-dsv05f45665-5885-40ad-d9m3:" ] } }, "defTrajectoryId": "<authority>:<source>:trajectory-sxs0f1a5219-4640-50af-9f63-c09140d57c12:", "wellId": "<authority>:well:external-REQxODI4MTAxMw:", "WellboreID" : "<authority>:Wellbore:external-TYOnPII4MTAxMw:" } }- CSV ingestion supports two kinds of relationships:
NOTE: In the final record, 'WellboreID' is resolved without the schema validation since we have a copyTo attribute in the related-natural-key block in the File Metadata. If the user misses adding copyTo attribute, then we go ahead and resolve the schema based on the x-osdu-relationship present in the target schema. In this case since there are two attributes, 'WellboreID' and 'VerticalReferenceEntityId', that are related to 'Wellbore', the result will be non-deterministic in the absence of copyTo attribute.
- In case of multiple parent records, the precendence is given to Master records, i.e, the records that have the following tag,
"DataMastering": "Master". However, if there's no Master parent record present, we put the first parent record returned by Search Service against the respective attribute.
- Spatial data
Pre-requisities:
The target schema should have an OSDU Spatial location block.
The CSV file has the Spatial data attributes.
The ExtensionProperties Block is used to provide the content details of the file. The Workflow service uses this same block to provide Spatial data information.
SpatialMapping: This section is used to create the Spatial data block in the ingested records.
- type: This field refers to the type of the Spatial data; currently the Workflow service only supports point.
- latitude: This field refers to the latitude of the point.
- longitude: This field refers to the longitude of the point.
{ "ExtensionProperties": { "FileContentsDetails": { "TargetKind": "<authority>:<source>:wellbore:1.0.0", "nestedFieldDelimiter": ".", "FileType": "csv", "SpatialMapping": { "type": "point", "latitude": "location.[0].latitude", "longitude": "location.[0].longitude" }, "FrameOfReference": [ { "kind": "CRS", "name": "GCS_WGS_1984", "persistableReference": "{\"wkt\":\"GEOGCS[\\\"GCS_WGS_1984\\\",DATUM[\\\"D_WGS_1984\\\",SPHEROID[\\\"WGS_1984\\\",6378137.0,298.257223563]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433],AUTHORITY[\\\"EPSG\\\",4326]]\",\"ver\":\"PE_10_3_1\",\"name\":\"GCS_WGS_1984\",\"authCode\":{\"auth\":\"EPSG\",\"code\":\"4326\"},\"type\":\"LBC\"}", "propertyNames": [ "location.[0].latitude", "location.[0].longitude", "location.[1].latitude", "location.[1].longitude" ], "propertyValues": [ "deg" ], "uncertainty": 0 } ] } } }
- AutoGuids :
- GUIDs can be used to track data transfer across the applications.
- Any attribute in OSDU record, which is supposed to have GUID value can be ingested using this feature.
- We can also use ExtensionProperties block in the file metadata record to specify that one or more properties/columns should be inserted auto-generated GUID strings for ingestion.
- The GUID string is inserted only when the respective column for which GUID has to be created is not present in CSV file or if the column is present in CSV file, it will append the GUID values in the records which are empty in CSV file for that respective column.
- This feature can be used to ingest GUIDs in a particular attribute. E.g., GUID is required in IntervalID attribute of osdu:wks:work-product-component--WellboreIntervalSet:1.2.0 record:
{ "ExtensionProperties": { "Name": "File", "FileContentsDetails": { "TargetKind": "kind" }, "GroupBy": ["UBHI", "Wellbore Name", "Name"], "AutoGuids": [ "Intervals[].IntervalID" ] } }
The ingestion framework comes with pre-registered DAGs to do CSV ingestion, and the ingestion framework created the corresponding workflows, which are available to use:
Multistep DAG:
- workflow name:
csv_parser_wf_status_gsm - workflow id:
Y3N2X3BhcnNlcl93Zg==
Implementation Note: Scope of Use
GroupBy is an SLB extension, which means that it is an SLB-specific functionality and not available in the community edition.
When you use Group By , CSV Parser does the following steps:
- Verify if Group By is enabled and has non-empty list of Group By headers
- Upload all the records (currently we are limiting number of records to 170k according to our performance tests results)
- Group records based on Group By headers values
- Transform grouped records (operation similar to flatmap/reduce), all the fields with
[].pattern will be collected into lists and the other fields will be taken as is(see examples below) - Pass transformed records to the list of handlers like we do with regular (not group by) case. Records are processed in parallel fashion during this step.
To start using this functionality you need to add 2 alterations to the regular CSV Parser's setup:
In file metadata which you specify during csv file upload you need to configure file's metadata that
CSV Parser can tell if you want it to use group by functionality and which CSV headers it should use for grouping. This should be done in the following json path "data.ExtensionProperties.GroupBy".
GroupBy field is a list of string values (case should be the same as in CSV headers).
Below you can see an example, note that some fields were omitted for simplicity of the illustration.
{
"data": {
"ExtensionProperties": {
"GroupBy": [
"Name",
"WellboreID"
]
}
}
}If is not there, or it is an empty list, CSV Parser will act as if there is no request for grouping, see an example below.
{
"data": {
"ExtensionProperties": {
"GroupBy": []
}
}
}Once you specified group by configuration in file's metadata CSV Parser will be grouping records based on that configuration. In other words all the records which have the same set of values for the same set of csv headers will fall into the same aggregation group (the same logic as you can see in SQL’s GROUP BY). The next items will explain which field values will be aggregated (grouped by) and how to deal with non grouped (common) attributes.
Implementation Note: Group By Header Values
If we have a group by column (header) without values these records will be skipped from the processing.
In order to add field to aggregation you need to add a special marker to that field [].. See an example below:
"Markers.MarkerName" --> "Markers[].MarkerName"
"Markers.MarkerMeasuredDepth" --> "Markers[].MarkerMeasuredDepth"
"Markers.MarkerDate" --> "Markers[].MarkerDate"As you can notice from the previous question CSV parser will be aggregating field values from headers with []. pattern in its name. This pattern allow user to specify different aggregation groups like Markers[].MarkerName and Markers2[].MarkerName will be aggregated into different lists. For example if we have 3 records fallen in the same group by header group values and this csv has 2 markers Markers[].MarkerName and Markers2[].MarkerName values will be grouped based on name on the left side of the pattern []. into Markers and Markers2 respectively. See example below:
{
"Markers": [
{
"MarkerName": "a"
},
{
"MarkerName": "b"
},
{
"MarkerName": "c"
}
],
"Markers2": [
{
"MarkerName": "d"
},
{
"MarkerName": "e"
},
{
"MarkerName": "f"
}
]
}The same logic applied within a single aggregation. If we have Markers[].MarkerName and Markers[].MarkerMeasuredDepth we will get something like this:
{
"Markers": [
{
"MarkerName": "a",
"MarkerMeasuredDepth": 100
},
{
"MarkerName": "b",
"MarkerMeasuredDepth": 200
},
{
"MarkerName": "c",
"MarkerMeasuredDepth": 300
}
]
}Common fields are the fields not identified by the []. pattern in their names. Those fields are supposed to have the same value between all the records from the same group. Those are the fields which does not fall neither in group by headers nor in aggregation fields. For such fields the first valid value (not empty) of each common column in a group will be taken as the value of that attribute for that grouped record created.
Implementation Note: Common Fields
Do not mistake common fields values with group by headers values. Here we are talking about fields which does not fall neither in group by headers nor in aggregation fields.
For such fields the first record in a group will be taken as source of values.
In order to provide yourself with some closer to reality examples, you can see the following screenshots.
Markers in CSV 
Markers in resulted Record 
Above you can see that UWI field was not aggregated - because it is a common field for all records in the group.
Implementation Note: File Size
The current implementation of Group By is limited to working with CSV files that do not exceed 170k rows. In case a user would try to use Group By functionality with a file bigger than 170k rows, CSV Parser will throw an error saying that mentioned limit is exceeded and such files are not supported by the Group By functionality.
Ingestion workflow service known limitations:
- Not yet integrated with OSDU Data Platform Notification service.
- Manually call the Workflow service to execute a registered DAG.
- Validation of DAGs & custom operators not yet built. You must make sure that the DAGs and custom operators passed as the payload are correct as per Airflow constructs.
- Scalability limitations: 90 queued up workflow requests with 10 concurrent workflows execution.
- Currently, only spatial point data (Latitude/Longitude or Y/X) is supported for the generation of the spatial location block.
- Only one key is used to search for the parent record in relationships that require search. Multiple keys support has not yet been implemented.
- Nested arrays are not supported for ID generation and relationships.
- Arrays without index attributes are not supported by any of the CSV handlers.
- A Metablock should not be created when all the attributes used in the propertyvalues are not present in the record.
- The Spatial Location block is partially created if the attributes used to create Spatial Location are missing in the schema.
- The CSV parser fails to create a relationship when the relationships in the metadata have the same entity.
- The attributes that hold the parent IDs used to create the relationship are listed in the data block of the ingested records.