Platform Management - Enterprise Integration Patterns on Azure

Management

When implementing modern application integration solutions, we rely on platforms that abstract many of the challenges inherent to these types of solutions. While the Enterprise Integration Patterns previously covered in this series describe architectural and implementation patterns, the Platform Management patterns focus on the operational aspects of an integration platform. In this post, we cover how the Platform Management patterns can be implemented on Azure.

This is the last post of the series describing how to implement the Enterprise Integration Patterns using the Azure Integration Services:

The patterns covered in this article are listed below.

Control Bus
Detour
Wire Tap
Message History
Message Store
Tracing Correlation Identifier (*)
Message Resubmission (*)
Repair and Resubmit (*)
Smart Proxy
Monitoring Events and Alerts (*)
Test Message
Channel Purger

Control Bus

Integration solutions are distributed in nature. Different components, including channels (pipes), processors (filters), and process managers, can reside in multiple networks or locations. To manage and support an integration solution, we need to understand not only what the different components are but also their health status, metrics, alerts, exception messages, etc. The Control Bus is meant to provide this visibility.

Implementation

	A custom Azure Dashboard can be created that includes charts, metrics, log queries, and links to the relating resources. Azure resource groups could also be used to group relating components.
	Serverless360 is a third-party tool that allows you to define composite applications for management and monitoring purposes.

Detour

The Detour pattern proposes that messages can be routed to an alternate path at run time using a Context-Based Router and toggled using the Control Bus. This alternate path might be a set of Pipes and Filters with additional steps.

Implementation

Event Grid

Context-based routing: Service Bus topics or Event Grid topics can be used to implement context-based routing.

Detour toggle: Azure Storage or any other persistence layer could be used to toggle the detour.

Azure Functions_COLOR

Context update: Logic Apps or Azure Functions can change the message context by updating the message header property used for routing based on the detour toggle state.

Wire Tap

When all messages that travel across a channel must be inspected for monitoring or troubleshooting, a Wire Tap can be implemented so a copy of all messages is sent to a secondary channel.

Implementation

	When using Service Bus topics, an additional subscription could be implemented so that a Polling Consumer persists all messages for monitoring or troubleshooting purposes.
	When using Event Grid topics, an additional subscription could be implemented so that an Event-Driven Consumer persists all messages for monitoring or troubleshooting purposes.

Message History

In some scenarios, where multiple Pipes and Filters process a message without having an orchestrating Process Manager, it can be difficult to troubleshoot a message without knowing what steps it has gone through. The Message History pattern suggests that history metadata can be added to a message as part of the Message Envelope.

One way to implement this pattern is to rely on the Pipes and Filters architectural style and add the details of each pipe and filter as history metadata to the Message Envelope in each filter.

Implementation

Logic Apps and Azure Functions can act as filters.

Service Bus can act as a pipe. The message history can be handled as a Service Bus message header or in a Message Envelope.

Event Grid can act as a pipe. The message history can be added to the Message Envelope.

Message Store

For monitoring, troubleshooting, or analysis purposes, we can use a Message Store to persist relevant information of the messages that are being processed by the integration platform. Depending on the business requirements, cost, storage space, and performance implications, metadata, the full payload, or both could be stored.

Implementation

Logic Apps stateful workflows persist all inputs and outputs of every action of the workflow. When dealing with sensitive data, obfuscation could be applied.

With Azure Functions, we can implement custom structured logging.

Tracing Correlation Identifier (*)

Previously, we discussed the Correlation Identifier pattern, which is useful to correlate messages in a Request-Reply implementation. However, when we are implementing the Message Store pattern across distributed components, it is important that we can correlate them. We can include a Tracing Correlation Identifier to be able to associate different log records and follow the processing flow of a single message across distributed Pipes and Filters. This pattern is not discussed in the Enterprise Integration Patterns, but I believe it is quite useful and common.

Implementation

	When using stateful workflow in Logic Apps, a Tracing Correlation Identifier is maintained automatically. You can add custom metadata to your tracing events as described here. Once you have implemented custom tracing events, then you can public custom queries and dashboards as described here.
	With Azure Functions and Application Insights, we can implement correlated structured logging or rely on the automatic correlation of Application Insights and leverage its timeline view.

Message Resubmission (*)

Previously, we described the Dead Letter Channel pattern, which provides a graceful way to remove and troubleshoot messages from the queue or topic subscription when these could not be delivered successfully due to issues not intrinsic to the message. When the reason for dead lettering was transient, these messages could be resubmitted into the main queue or topic for a retry. This pattern is not described in the Enterprise Integration Patterns book, however, it is quite common in integration solutions.

Implementation

Regardless of the Messaging Channel in use, explore the option of resending the message from the source when possible. With this approach, the dead-lettered message must then be removed as a separate operation.

When using Service Bus as the Messaging Channel, we can use different tools for resubmitting a message from the dead-letter sub-queue to the main queue or topic.

Service Bus Explorer (Windows-based client): It is a free Windows tool that allows resubmitting a message in a dead-letter queue. More information on the documentation.

Portal-based Service Bus Explorer - It supports data operations on the queues, topics, and subscriptions (and their dead letter sub-entities) from the Azure portal. A message can be read from the dead-letter queue and copied to generate a new message to the queue or topic.

When resubmitting a message, the considerations below must too be taken into account:

If duplicate detection is enabled, a new MessageId system property might be required depending on the duplicate detection history configuration.

For topics, when a message is resubmitted, all subscription filters would get a copy of the message. To avoid sending redundant messages to non targeted receivers, the appropriate idempotence mechanisms or message metadata and filters must be in place.

Repair and Resubmit (*)

When a message has been dropped to the Invalid Message Channel, it means that it cannot be processed as is; it requires some updates before it is being resubmitted. If the message cannot be fixed on the source system and resent, we need to explore the option of repairing and resubmitting the message on the integration platform. This pattern is not discussed in the Enterprise Integration Patterns, but worth exploring.

Implementation

When using Service Bus as the Messaging Channel, we can use different tools for repairing and resubmitting a message in the dead-letter sub-queue.

Service Bus Explorer (Windows-based client): As discussed above, this tool allows resubmitting a message in the dead-letter queue. Once the message has been selected, the message body, system properties, and custom properties can be edited before being resubmitted. More information on the documentation.

Portal-based Service Bus Explorer – As described above, it allows reading messages from the dead-letter queue and copy them to generate and submit a new message to the queue or topic.

See the considerations in the previous section when resubmitting messages.

Smart Proxy

The Smart Proxy pattern describes an implementation of the Request-Reply pattern where multiple requestors can send requests and expect responses to their own independent queues, while the replier expects to have a single queue for all requests and responses. The Smart Proxy is a set of Pipes and Filters in which a pipe is in charge of translating the Return Address so that the replier can work with just a pair of queues, while the requestors keep their own.

Implementation

Service Bus queues can act as pipes

Azure Functions or Logic Apps can act as filters. They would require a persistence layer to be able to map the Return Address.

Monitoring Events and Alerts (*)

When managing an integration platform, we often want to be notified when certain events occur. For instance, an integration processed failed, there are messages in a dead-letter sub-queue, etc. This pattern is not described in the Enterprise Integration Pattern. However, I believe it is noteworthy.

Implementation

	When using Logic Apps, monitoring alerts can be implemented.
	When using Azure Functions, monitoring alerts can be implemented using Azure Monitor or Application Insights.
	When using Service Bus, metrics can be used to establish alerts, including dead-lettered message count.

Test Message

The Test Message pattern suggests that when we need to test integration solutions end-to-end, we can design our solution in a way that we can inject test messages in a pipe, let the test message go through the series of Pipes and Filters, and monitor the test message at the end without impacting the receiver applications.

Implementation

	Service Bus can be used as a Messaging Channel or pipe. Metadata must be added in the Message Header or Envelope Wrapper to signal all Pipes and Filters when a message is a test. The last pipe should be a topic, in which the subscription for the receiver does not consider test messages and a test subscription receives only test messages.
	Event Grid can be used as a Messaging Channel or pipe. Metadata must be added in the Envelope Wrapper to signal all Pipes and Filters when a message is a test. The last pipe should be a topic in which the subscription for the receiver does not consider test messages and a test subscription forwards only test messages.

Channel Purger

The Channel Purger pattern suggests that a Messaging Channel should provide tools or means to be purged. This might not be recommended in production environments, but it is useful when developing or testing integration solutions.

Implementation

	When using Service Bus, the Windows-based client Service Bus Explorer can be used to purge queues, dead-letter sub-queues or topic subscriptions. Additionally, when possible a queue or topic subscription can be deleted and recreated.
	When using Event Grid, a topic could be deleted to be purged.

Wrapping up

In this post, we have covered the Platform Management of the Enterprise Integration Patterns and how to leverage Azure to implement them. With this post, the series comes to an end. I hope you have found the whole series useful and these posts have provided some handy information for when you are architecting or implementing integration solutions on Azure. Thanks for reading and making it this far!

Happy integration!

Cross-posted on Deloitte Engineering
Follow me on @pacodelacruz