Messaging Transformation - Enterprise Integration Patterns on Azure

It is quite common that integration solutions interact with diverse applications, each with disparate data models and heterogeneous formats. In this post, we will discuss the Message Transformation patterns and how these can be implemented on Azure to overcome the challenges derived from working with different message structures and representations.

This post is a part of a series describing how to implement the Enterprise Integration Patterns using the Azure Integration Services:

1. Introduction
2. Message Construction
3. Messaging Channels
4. Messaging Endpoints
5. Message Routing
6. Message Transformation (this)
7. Platform Management

The patterns covered in this article are listed below.

• Message Translator
• Envelope Wrapper
• Content Enricher
• Content Filter
• Claim-Check
• Normaliser
• Canonical Data Model
• Compression/Decompression (*)
• Encryption/Decryption (*)

Message Translator

The Message Translator pattern describes how data can be translated from one format to another. Transformation can happen at different levels:

• Data structure: related to entity hierarchies, relationships, cardinality, etc.
• Data types: related to cross-references, domains, constraints, data types, field names, etc.
• Data representation: deals with the format, e.g. binary, JSON, XML, CSVs, key-value pairs; industry standards like EDIFACT, X12; vendor-specific formats, character sets, encryption, compression, digital signatures, etc.
• Transport: deals with communication protocols such as HTTP, JMS, AMQP, gRPC, SOAP, etc.

Implementation

There are different ways to transform messages in Logic Apps For data structure and data type transformations: ·         Simple JSON object transformation with built-in actions ·         Advanced JSON object transformations with Liquid templates ·         Transform XML to XML For data representation transformations: ·         XML to JSON ·         JSON to XML ·         JSON array to CSV ·         JSON array to HTML ·         Conversion functions ·         Flat file encoding and decoding ·         EDI X12 encoding and decoding ·         EDIFACT encoding and decoding ·         RosettaNet encoding and decoding

On Azure Functions, custom transformations can be done via custom code.

Envelope Wrapper

The Envelope Wrapper pattern outlines how integration solutions can support adding a wrapper to messages so they can contain metadata required by the integration process while being able to deliver messages to the intended receivers in the format they need.

Implementation

	On Logic Apps, an Envelope Wrapper can easily be added and removed using the Compose built-in action as shown here or leveraging Liquid templates for more complex scenarios.
	On Azure Functions, Envelope Wrappers can be added and removed via custom code.

Content Enricher

The Content Enricher pattern is useful when the message coming from a source system does not contain all the data required to be processed in the target system. This filter can access an additional data source to augment the message with the missing information.

Implementation

	Logic Apps connectors can be used to retrieve the missing information from the enrichment source(s).
	Azure activity functions can be used to retrieve the missing information from the enrichment source(s) in an integration process orchestrated by Durable Functions.

Content Filter

The Content Filter pattern describes how to remove data elements or hierarchies that are not relevant to the target system. This pattern can also be used to remove sensitive information before a message is delivered to other systems.

Implementation

	See the Message Translator implementation section above related to data structure and data type transformations.
	On Azure Functions, custom transformations can be done via custom code.

Claim-Check

The Claim-Check pattern is helpful when the message size must be reduced without losing information. This pattern suggests that the full message can be stored in a durable store and a claim can be passed to subsequence components. Those components would use the claim to retrieve the message persisted.

When the main limitation is the Message Channel’s message size limitation, consider using the Compression/Decompression (*) pattern when applicable to avoid managing another component for storage and the lifecycle of the persisted messages.

Implementation

	This pattern can be implemented on Logic Apps as described here.
	This pattern can be implemented on Azure Functions using custom code.

Normaliser

The Normaliser is a special type of Message Translator that translates semantically equivalent messages but with different structures to a common format. The Normaliser is a combination of a Message Router and a set of Message Translators.

Implementation

The implementation of each of the part of this composed pattern has been described previously.

Canonical Data Model

The Canonical Data Model is beneficial when messages semantically equivalent are to be interchanged between multiple applications, each with disparate formats or structures. The value of a Canonical Data Model is realised when the number of Message Translators can be reduced by having a common model or when decoupling from proprietary formats is important.

Implementation

	See the Message Translator implementation section above related to data structure and data type transformations.
	On Azure Functions, custom transformations can be done via custom code.

Compression/Decompression (*)

The Compression/Decompression (*) pattern is a type of data representation transformation. While this is not mentioned as a pattern in the Enterprise Integration Patterns book, I believe it is important highlighting it, as it is a quite common requirement. When dealing with large messages, I prefer to use compression instead of the Claim-Check pattern when possible because it is typically much simpler to implement.

Implementation

Compression and decompression can be implemented using custom code on Azure Functions.

Encryption/Decryption (*)

The Encryption/Decryption (*) pattern is another kind of data representation transformation. Although it is not mentioned as a pattern in the Enterprise Integration Patterns book, it is a quite common requirement.

Encryption is usually required for data in transit and data at rest. For encryption at rest, encryption can be either client-side or server-side. Client-side encryption means that encryption is performed in memory by an application before the data is being persisted. Server-side encryption at rest is when the storage service performs encryption/decryption transparently when storing or retrieving the data.

Implementation

	Service Bus provides inbuilt server-side encryption at rest. All requests to Service Bus are protected using encryption in transit.
	Event Grid provides inbuilt server-side encryption at rest. All requests to Event Grid are protected using encryption in transit.
	For scenarios when client-side encryption is required, Logic Apps supports encrypting data with the “Encrypt data with key” action of the Key Vault connector.
	For scenarios when client-side encryption is required, Azure Functions can encrypt messages using custom code and Key Vault encryption.

Wrapping Up

In this post, we have covered the Messaging Transformation patterns and how to leverage the Azure Integration Services to implement them. Understanding these patterns allows us to consider different approaches when architecting integration solutions that require dealing with messages with disparate structures and different formats. I hope you have found this post useful. Stay tuned to the next and last instalment of this series covering the Platform Management patterns.

Happy integration!

Cross-posted on Deloitte Engineering
Follow me on @pacodelacruz