• Introduction
• Terminology
• Mapping definition highlights
• Transformation flow
• Sample - Raw to WKS record snippet
• When adding a new data partition
• Supported mapping operations
• Default operations
• Relationship block transformation scenarios
• Meta block transformation examples
• Publish the WKS record creation status

The WKS Transformation service transforms data from its original structure into a Well Known Structure, commonly referred to as WKS. It is an internal service (not exposed for external calls) and reacts to original records being stored or updated in the system.

Transformations performed by the WKS Transformation service are intended to be structural transformations only, so the values of the data are not changed in any way. Units are not converted, dates are not formatted, etc. All original values are retained.

Data clipping may occur, depending on the source schema and the target schema. For example, if the source schema has 100 attributes, and the target WKS schema has only 25 attributes, then 75 attributes would be "clipped" (omitted) in the WKS record. Regardless, the original data is always available in the original record.

The ancestry data block of the WKS record holds the ID of the parent record from which the WKS was created. The completeness of the data depends on the schema and the mapping definition.

Note: 'Transformations of a raw record to a WKS record happen in the sequential order of the operations provided in a mapping file. It is important that the operations in the mapping file are provided in the correct order.`

• Original or source schema: Refers to the schema (structural representation) of the record or data that is ingested.
• Target schema: Refers to the WKS schema into which the original data is transformed.
• Mapping definition: Definitions that map attributes from the original schema to the target schema.

• The mapping definition contains the target schema kind information, such as osdu:wks:master-data--Wellbore:*.*.*, and lists of mappings for all source attributes to its corresponding target attribute.
• There must be a separate mapping definition for every source schema to target schema kind pair, and they must reside in the data partition.

For records ingested or updated in the Data Platform, the Storage service publishes an event named a 'Storage event'. The WKS Transformation service reacts to this event, and if applicable, executes a transformation process as outlined below:

• The WKS Transformation service picks a storage event.
• It checks and skips transformation if any or all of the following conditions are true.
  • The record is already a WKS or WKE record. The source of the record kind is WKS or WKE.
  • The entity of the raw record kind is any of following: 'work-product-component--Document', 'DataSet--GenericDocumentArtefact', 'DataSet--Page'.
• Based on the original record kind, it looks for a corresponding mapping definition in the system.
• If a mapping definition exists, the system:
  • Fetches the mapping definition.
  • Retrieves the original record from storage.
  • Uses the mapping information and transforms the original record to a WKS record.
  • Transforms the data block per mappings defined in the mapping definition.
  • Transforms meta, tags, acl, legal, and relationship blocks of a raw record by default without any explicit operation.
  • The kind of the WKS record is the same as the target kind defined in the mapping definition for the given source kind.
  • Ingests the WKS record back into the Data Platform using the Storage service.
• If a mapping does not exist, then transformation does not occur.

The WKS service does not start listening to events from a newly provisioned data partition automatically. The service must be re-deployed to start using this new data partition.

The WKS Transformation service currently supports the following mapping operation types:
COPY, COPY_SUBSTRUCTURE, RELATIONSHIP, RELATIONSHIP_ARRAY, CONSTANT, AGGREGATE, CONCATENATE, COPY_ARRAY_INDEX, CONDITIONAL_PRIORITY, and REFERENCE_LOOKUP.

For the complete set of supported mapping operations and their usage, refer to the Mapping Service tutorial

The WKS transformation service performs some operations by default. These operation are performed even if they are not defined in the mapping definition.

Raw records, called 'parentRecord' in this document, maintain relationships with other record or records, called 'relatedRecord' in this document, using relationship blocks. WKS records generated from such a parentRecord hold the relationship with the WKS records of the relatedRecord.

The following diagram represents this scenario:

The following table shows the behavior of the WKS record's relationship block based on the availability of the relatedRecord, the mapping definition, and the WKS of the relatedRecord:

Note: A reference to only a single WKS record for a given relatedRecord is included in the relationship block of the parentRecord WKS. For the situation where there are multiple WKS records present for a given relatedRecord, the selection of a single WKS record out of these WKS records is done based on the following criteria:

• The WKS record has the same authority as that of the relatedRecord.
• The WKS record has a higher major version. This would be the case if a mapping exists for more than 1 major version of the WKS schema.

If a property path in the “propertyNames” array in the meta block of a raw record is defined as “sourceProperty” in a COPY mapping operation, and that mapping operation is successful, then corresponding target path, with the resolved value # if any, will be added to “propertyNames” in meta block of the generated WKS.

{
"kind": "CRS", "name": "GCS_WGS_1984", "persistableReference": "persistableReferenceValue", "propertyNames": [ "wellHeadProjected.x" ] }

{ "mappings": [ { "identifier": "data.SpatialLocation", "operations": [ { "type": "COPY", "sourceProperty":"data.wellHeadProjected.x", "targetProperty":"data.Location.features.geometry.coordinates[#].lat " } ] } ] }

{ "kind": "CRS", "name": "GCS_WGS_1984", "persistableReference": "persistableReferenceValue", "propertyNames": [ "Location.features.geometry.coordinates[0].lat" ] }

{ "meta": [ { "kind": "CRS", "name": "GCS_WGS_1984", "persistableReference": "persistableReferenceValue", "propertyNames":[ "SpatialLocation.features.geometry.coordinates[0].x", "SpatialLocation.features.geometry.coordinates[1].x", "SpatialLocation.features.geometry.coordinates[2].x" ], "propertyValues": [ "deg" ] } ] }

{ "mappings": [ { "identifier": "data.SpatialLocation", "operations": [ { "type": "COPY", "sourceProperty": "data.SpatialLocation.features.geometry.coordinates[].x", "targetProperty": "data.Location.features.geometry.coordinates[].x" } ] } ] }

{ "meta": [ { "kind": "CRS", "name": "GCS_WGS_1984", "persistableReference": "persistableReferenceValue", "propertyNames": [ "Location. features.geometry.coordinates[0].x", "Location. features.geometry.coordinates[1].x", "Location. features.geometry.coordinates[2].x" ], "propertyValues": [ "deg" ] } ] }

{ "meta": [ { "kind": "CRS", "name": "GCS_WGS_1984", "persistableReference": "persistableReferenceValue", "propertyNames": [ "SpatialLocation.features.geometry.coordinates" ] "propertyValues": [ "deg" ] } ] }

{ "mappings": [ { "identifier": "data.SpatialLocation", "operations": [ { "type": "COPY_SUBSTRUCURE", "sourceProperty": "data.SpatialLocation", "targetProperty": "data.Location " } ] } ] }

{ "meta": [ { "kind": "CRS", "name": "GCS_WGS_1984", "persistableReference": "persistableReferenceValue", "propertyNames": [ "Location.features.geometry.coordinates" ], "propertyValues": [ "deg" ] } ] }

{ "meta": [ { "kind": "CRS", "name": "GCS_WGS_1984", "persistableReference": "persistableReferenceValue", "propertyNames": [ "SpatialLocation.features.geometry.coordinates" ], "propertyValues": [ "deg" ] } ] }

{ "mappings": [ { "identifier": "data.SpatialLocation", "operations": [ { "type": "COPY", "sourceProperty": "data.SpatialLocation.features.geometry.coordinates[]", "targetProperty": "data.Location. features.geometry.coordinates[]" } ] } ] }

{ "meta": [ { "kind": "CRS", "name": "GCS_WGS_1984", "persistableReference": "{persistableReference}", "propertyNames": [ " Location.features.geometry.coordinates" ], "propertyValues": [ "deg" ] } ] }

The WKS service publishes the status of WKS record creation to statuschangetopic. It can be used to track the status of the transformation of a raw record to a WKS record. statuschangedtopic is a pub/sub based topic that is configured in the cloud. Consumer services can subscribe to this topic to receive status change events.

• Status - Status holds the status of the raw record transformed to a WKS record.

The WKS service only publishes Status messages.

The following are the types of status details that the WKS service publishes:

For raw records, the recordIdVersion and userEmail cannot be stored because the WKS service does not have that information in the record changed event.

When transformation starts, a raw record ID is sent as recordId. When the WKS is successfully stored, a WKS ID is sent as a recordId. In additionalProperties the raw record ID and version are sent as inputRecordId and inputRecordIdVersion.
In each case, a corresponding message is provided:

The WKS service propagates the error messages that it receives as an error from the Storage service when storing the WKS record fails or has an error.

The WKS service can publish its own error message in the failure status when there is a failure in the WKS service.

When a WKS record fails, a raw record ID is sent because the recordId for the WKS record was not created.

For other possible error codes, see the Error Code table.

The WKS service propagates the error message that it receives as an error from the Mapping service when the mapping is not available. When the mapping is not found, the recordId for the WKS record is not created and a raw recordId is sent with a status message with the raw record ID.