Microsoft Azure Integration Platform as a Service (iPaaS) – Logic Apps and its Azure allies [Update]
Introduction
A year ago, I wrote a post about this very same topic, the Microsoft Azure Integration Platform as a Service (iPaaS). At that time, the core iPaaS product offering, Azure Logic Apps, was roughly 6 months old, since it’s generally available launch. Since the launch date, we’ve seen an impressive release cadence from the product team, an ever-growing list of connectors from other Microsoft product teams and third-party providers, and a considerable growth of the user-base.
In this post, I’ll talk about the current state of Azure Logic Apps and the other Azure services which are relevant when building application and process integration solutions. It’s worth mentioning that there is another platform which targets hybrid data integration solutions called Azure Data Factory, however, that’s an ETL platform, and won’t be discussed in this post.
What's an Integration Platform as a Service (iPaaS)
According to Gartner, an “Integration Platform as a Service (iPaaS) is a suite of cloud services enabling development, execution and governance of integration flows connecting any combination of on premises and cloud-based processes, services, applications and data within individual or across multiple organizations.” Its capabilities usually include:
- Communication Protocol Connectors (HTTP, SFTP, AS2, etc.)
- Application connectors, including SaaS and on-premises apps
- Ability to handle data formats, like JSON or XML, and data standards like EDIFACT, HL7, etc.
- Orchestration
- Routing
- Data validation and transformation
- Monitoring and Management tools
- Full life-cycle API Management
- Development and solution life-cycle management tools
Microsoft’s core iPaaS product offering is Azure Logic Apps. However, we could argue that one of the key differentiators and advantages of Microsoft within the iPaaS market, is how Logic Apps is enriched with the whole Azure ecosystem, which also keeps getting richer and better.
Azure Services to Build Integration Solutions
Based on Gartner’s definition of iPaaS, I’ll describe the different Azure services which we can leverage to develop, implement and manage enterprise-class integration solutions. The figure below, shows the different capabilities of an iPaaS, and which Azure product offerings we can use to implement them.
Orchestration, Data Handling and Transformation, and Routing.
Logic Apps workflows allow us to orchestrate our integrations solutions. They provide a workflow visual designer (via the Azure Portal or Visual Studio) to design, build and manage our integration workflows. These workflows are serverless, which means that we can focus on the functionality and business value, while all the infrastructure and scaling are fully abstracted and we pay only for what we use. With Logic Apps we can implement long running workflows, as described in previous posts using the Webhook action, and the Service Bus connector. A figure of a Logic App workflow, which implements some conditions, a loop and some connectors, is shown below.
Behind the scenes, Logic Apps workflows are based on the Workflow Definition Language. Within the Logic App, we can validate messages and transform messages using Data Operations, Liquid, or XSLTs for XML documents. For Routing, we can use different Logic Apps features to control the workflow flow, and we can also use Azure Service Bus Topics Subscriptions for Pub/Sub scenarios.
Additionally, we can externalise Business Rules from Logic Apps workflows.
Connectors
In addition to the Logic Apps workflows, the ever-growing list of connectors, is another of the great features of this platform. These connectors allow us to trigger workflows and get data from and push data to many diverse apps on the cloud and on-premises through different protocols. Additionally, the on-premises data gateway, allows hybrid connectivity for some of the connectors. Below there is a snapshot of the 170+ connectors available at the time of writing. Please bear in mind that this list is always changing and that some connectors are still in preview and might not be available in all regions.
Protocol Connectors
|
Hybrid Connectors
|
Azure Connectors
|
Other Microsoft Connectors
|
Third-Party SaaS Connectors
|
|
A key component of Logic Apps connectivity is the on-premises data gateway, which allows the connectors listed above as “Hybrid” to connect to on-premises resources. The data gateway is an agent that can be installed on a server on-premises or on a VM on an Azure VNET. All the interchanges between the data gateway and Logic Apps are outgress (from on-premises to Azure) via a managed Service Bus Relay with encrypted channels. The data gateway can be installed on more than one VM to provide high-availability.
Serverless Compute (Custom Code and Custom Connectors)
As shown in the previous sections, Logic Apps provide a lot of connectors and functionality for message processing, however, in some scenarios, we might need to write custom code. With Azure Functions, we can write custom nano-services in C#, F#, Javascript or any other of the supported languages, and we can invoke those functions from our Logic Apps synchronously via HTTP or asynchronously via Azure Service Bus or Azure Event Grid, as described below.
We can use Azure Functions as part of our custom message processing or to send or receive messages to or from an application for which there is no out-of-the-box connector.
Additionally, we can use Azure Functions Hybrid connections to securely connect to resources on-premises via outbound calls to Azure Service Bus Relay.
Messaging and Eventing
Integration solutions usually require a way to send and receive messages in an asynchronous way. Additionally, modern computing is all about events, and increasingly, integration solutions have to react to or push events.
Azure Service Bus Messaging provides reliable message queuing and publish-subscribe messaging capabilities. Azure Service Bus Queues and Topics allow us to decouple in time upstream and downstream systems or different interrelated services. Service Bus Queues provide ordered message delivery to competing consumers, while Service Bus Topics enable publish-subscribe messaging scenarios where multiple consumers can process the same message.
The following Service Bus features make it a key component of most integration solutions on Azure:
- temporal decoupling
- message processing load balancing (competing consumers)
- Publish/Subscribe pattern via Service Bus Topics;
Azure Event Grid is a fully-managed event-routing platform on Azure which we can leverage as part of our integration solutions. Logic Apps can react to events on Event Grid coming from the different publishers or publish events for other handlers to consume.
While Service Bus provides rich and robust messaging capabilities, like transactions, ordering, sessions, dead-lettering, enqueue time, long time to live and deduplication, among others; Event Grid offers hyper-scale event routing, with filtering, routing, built-in Azure publishers, with a push-push model, but a short time to live and no ordering. Additionally, due to these limitations, Event Grid is not meant for critical or transactional messages, but for events, which might still point to its source, e.g. the event of a blob being created containing the URL of the actual blob.
API Mediation and Management
API Mediation and Management is a core funcitonality of integration solutions which require to expose RESTful APIs.
Azure API Management is an Azure service which functions as a API gateway for backend APIs in the cloud or on-premises. Some of the benefits of API Management are:
- Securing APIs with different authentication and authorisation methods, like Azure AD or use of client certificates
- A developer’s portal to publish APIs and speed up the on-boarding process
- Managing consumer accounts
- VNET connectivity to connect to other secured resources and on-premises APIs
- Caching to improve API response times
- Throttling requests
- Routing or forwarding
- Transforming requests, by changing the method or implementing richer transformation policies
- Tracing calls, or logging requests via Event Hubs
- Monitoring and analysing usage and health of APIs
- Mocking responses
- Monetising APIs
Furthermore, Azure Functions Proxies provide a small subset of what API Management does, which can be leveraged as a light API Gateway for HTTP-triggered Logic Apps, including:
- Azure Functions Proxies can be secured in a very similar way to App Services.
- Modify requests and responses, which also allows us to mock APIs.
- Consolidating multiple and disperse APIs into simpler URL routes.
Depending on the level of features we require, we can go with the lightweight Azure Functions Proxies or with the full API Management for our integration solutions.
Monitoring and Management
Azure Logic Apps provide built-in monitoring tools that allow you to check the run history, trigger history, status, performance, etc. You can also install the Logic Apps Management Solution on OMS, which gives you a very rich aggregated view and charts of all your logic apps that are being monitored. The OMS Logic Apps Management Solution view is shown in the figure below.
Additionally, you can build custom activity monitoring dashboards and publish custom queries based on diagnostics sent to Azure Log Analytics. More information on my previous posts:
- Business Activity Monitoring on Azure Logic Apps with Azure Log Analytics and
- Publishing Custom Queries of Logic Apps Execution Logs
A sample of what you can build in the figure as follows.
Development and Solution Life-cycle Management
To develop some of the iPaaS Solution components we can use the Azure Portal, however, when we think about the whole CI/CD pipeline, it’s better to work with Visual Studio and with Visual Studio Team Services (VSTS) for source control, build and release defections. More information about CI/CD for Logic Apps on my previous post:
Handy References
If you want to know more about Logic Apps, you might find the following links handy
Wrapping Up
In this post, I’ve discussed the current state of the Microsoft Azure Integration Platform as a Service (iPaaS) components, being Logic Apps the main player and complemented by other Azure product offerings. As mentioned previously, one of the main advantages of Microsoft in the iPaaS market is the whole Azure ecosystem and how we can build integration solutions leveraging the capabilities of all these different platforms. And we need to consider not only the technologies described here, but also all those which can be utilised through the connectors; such as Azure Cognitive Services and Azure Machine Learning.
Additionally, many of these components offer per-execution price or affordable entry-level pricing options. This enables us to start small when building enterprise-class integration solutions and grow from there.
I hope you’ve found this post useful, and please feel free to ask any questions or add your comments below.
Happy clouding!
Follow me on @pacodelacruz
Cross-posted on Deloitte Engineering Blog