Introduction

A year ago, I wrote a post about this very same topic, the Microsoft Azure Integration Platform as a Service (iPaaS). At that time, the core iPaaS product offering, Azure Logic Apps, was roughly 6 months old, since it’s generally available launch. Since the launch date, we’ve seen an impressive release cadence from the product team, an ever-growing list of connectors from other Microsoft product teams and third-party providers, and a considerable growth of the user-base.

In this post, I’ll talk about the current state of Azure Logic Apps and the other Azure services which are relevant when building application and process integration solutions. It’s worth mentioning that there is another platform which targets hybrid data integration solutions called Azure Data Factory, however, that’s an ETL platform, and won’t be discussed in this post.

What's an Integration Platform as a Service (iPaaS)

According to Gartner, an “Integration Platform as a Service (iPaaS) is a suite of cloud services enabling development, execution and governance of integration flows connecting any combination of on premises and cloud-based processes, services, applications and data within individual or across multiple organizations.” Its capabilities usually include:

  • Communication Protocol Connectors (HTTP, SFTP, AS2, etc.)
  • Application connectors, including SaaS and on-premises apps
  • Ability to handle data formats, like JSON or XML, and data standards like EDIFACT, HL7, etc.
  • Orchestration
  • Routing
  • Data validation and transformation
  • Monitoring and Management tools
  • Full life-cycle API Management
  • Development and solution life-cycle management tools

Microsoft’s core iPaaS product offering is Azure Logic Apps. However, we could argue that one of the key differentiators and advantages of Microsoft within the iPaaS market, is how Logic Apps is enriched with the whole Azure ecosystem, which also keeps getting richer and better.

Azure Services to Build Integration Solutions

Based on Gartner’s definition of iPaaS, I’ll describe the different Azure services which we can leverage to develop, implement and manage enterprise-class integration solutions. The figure below, shows the different capabilities of an iPaaS, and which Azure product offerings we can use to implement them.

01 iPaaS Components

Orchestration, Data Handling and Transformation, and Routing.

Logic Apps workflows allow us to orchestrate our integrations solutions. They provide a workflow visual designer (via the Azure Portal or Visual Studio) to design, build and manage our integration workflows. These workflows are serverless, which means that we can focus on the functionality and business value, while all the infrastructure and scaling are fully abstracted and we pay only for what we use. With Logic Apps we can implement long running workflows, as described in previous posts using the Webhook action, and the Service Bus connector. A figure of a Logic App workflow, which implements some conditions, a loop and some connectors, is shown below.

02 workflow

Behind the scenes, Logic Apps workflows are based on the Workflow Definition Language. Within the Logic App, we can validate messages and transform messages using Data OperationsLiquid, or XSLTs for XML documents. For Routing, we can use different Logic Apps features to control the workflow flow, and we can also use Azure Service Bus Topics Subscriptions for Pub/Sub scenarios.

Additionally, we can externalise Business Rules from Logic Apps workflows.

Connectors

In addition to the Logic Apps workflows, the ever-growing list of connectors, is another of the great features of this platform. These connectors allow us to trigger workflows and get data from and push data to many diverse apps on the cloud and on-premises through different protocols. Additionally, the on-premises data gateway, allows hybrid connectivity for some of the connectors. Below there is a snapshot of the 170+ connectors available at the time of writing. Please bear in mind that this list is always changing and that some connectors are still in preview and might not be available in all regions.

Protocol Connectors
  • FTP
  • HTTP
  • HTTP with Azure AD
  • HTTP with Swagger
  • RSS
  • SFTP
  • SMTP
  • SOAP-to-REST
  • WebHook
  • AS2
Hybrid Connectors
  • BizTalk
  • DB2
  • File System
  • Informix
  • MQ
  • MySQL
  • Oracle DB
  • PostgreSQL
  • REST
  • SAP
  • SharePoint
  • SOAP (to REST and pass-through)
  • SQL Server
  • Teradata
Azure Connectors
  • Azure AD
  • Azure API Management
  • Azure App Services
  • Azure Application Insights
  • Azure Automation
  • Azure Blob Storage
  • Azure Container Instance
  • Azure Data Lake
  • Azure Data Factory
  • Azure Event Grid
  • Azure File Storage
  • Azure Functions
  • Azure Kusto
  • Azure Logic Apps
  • Azure ML
  • Azure Resource Manager
  • Azure Security Center
  • Azure SQL Data Warehouse
  • Azure Storage Queues
  • Azure Table Storage
  • Computer Vision API
  • Common Data Service
  • Content Moderator
  • Cosmos DB
  • Custom Vision
  • Event Hubs
  • Face API
  • LUIS
  • QnA Maker
  • Service Bus
  • SQL Server
  • Text Analytics
  • Video Indexer
Other Microsoft Connectors
  • Bing Maps
  • Bing Search
  • Dynamics 365
  • Dynamics 365 for Financials
  • Dynamics Nav
  • Microsoft Forms
  • Microsoft Kaizala
  • Microsoft StaffHub
  • Microsoft Teams
  • Microsoft To-Do
  • Microsoft Translator
  • MSN Weather
  • Office 365 Excel
  • Office 365 Groups
  • Office 365 Outlook
  • Office 365 Video
  • OneDrive
  • OneDrive for Business
  • OneNote
  • Outlook Customer Manager
  • Outlook Tasks
  • Outlook.com
  • Project Online
  • Power BI
  • SharePoint
  • Skype for Business
  • VSTS
  • Yammer
Third-Party SaaS Connectors
  • 10to8
  • Adobe Creative Cloud
  • Apache Impala
  • Appfigures
  • Asana
  • Aweber
  • Basecamp3
  • Benchmark Email
  • Bitbucket
  • Bitly
  • Blogger
  • Box
  • Buffer
  • Calendly
  • Campfire
  • Capsule CRM
  • Chatter
  • Cognito Forms
  • D&B Optimizer
  • Derdack Signl4
  • DocFusion
  • Docparser
  • DocuSign
  • Dropbox
  • Easy Redmine
  • Elastic Forms
  • Enadoc
  • Eventbrite
  • Facebook
  • FlowForma
  • FreshBooks
  • Freshdesk
  • Freshservice
  • GitHub
  • Gmail
  • Google Calendar
  • Google Drive
  • Google Sheets
  • Google Tasks
  • GoToMeeting
  • GoToTraining
  • GoToWebinar
  • Harvest
  • HelloSign
  • HipChat
  • iAuditor
  • Infobip
  • Infusionsoft
  • Inoreader
  • insightly
  • Instagram
  • Instapaper
  • Intercom
  • Jira
  • JotForm
  • Kintone
  • LeanKit
  • LiveChat
  • Lithium
  • MailChimp
  • Mandrill
  • Marketing Content Hub
  • Metatask
  • Muhimbi PDF
  • MySQL
  • Nexmo
  • Oracle Database
  • Pager Duty
  • Parserr
  • Paylocity
  • Pinterest
  • Pipedrive
  • Pitney Bowes Data Validation
  • Pivotal Tracker
  • Planner
  • Plivo
  • Plumsail Documents
  • Plumsail Forms
  • Plumsail SP
  • PostgreSQL
  • Redmine
  • Salesforce
  • SendGrid
  • ServiceNow
  • Slack
  • Smartsheet
  • SparkPost
  • Stripe
  • SurveyMonkey
  • Tago
  • Teamwork Projects
  • Teradata
  • Todoist
  • Toodledo
  • Trello
  • Twilio
  • Twitter
  • Typeform
  • UserVoice
  • Vimeo
  • WebMerge
  • WordPress
  • Workday HCM
  • Workday Finance
  • Wunderlist
  • YouTube
  • Zendesk
  • Zoho

A key component of Logic Apps connectivity is the on-premises data gateway, which allows the connectors listed above as “Hybrid” to connect to on-premises resources. The data gateway is an agent that can be installed on a server on-premises or on a VM on an Azure VNET. All the interchanges between the data gateway and Logic Apps are outgress (from on-premises to Azure) via a managed Service Bus Relay with encrypted channels. The data gateway can be installed on more than one VM to provide high-availability.

03 On Prem data Gateway

Serverless Compute (Custom Code and Custom Connectors)

As shown in the previous sections, Logic Apps provide a lot of connectors and functionality for message processing, however, in some scenarios, we might need to write custom code. With Azure Functions, we can write custom nano-services in C#, F#, Javascript or any other of the supported languages, and we can invoke those functions from our Logic Apps synchronously via HTTP or asynchronously via Azure Service Bus or Azure Event Grid, as described below.

We can use Azure Functions as part of our custom message processing or to send or receive messages to or from an application for which there is no out-of-the-box connector.

Additionally, we can use Azure Functions Hybrid connections to securely connect to resources on-premises via outbound calls to Azure Service Bus Relay.

Messaging and Eventing

Integration solutions usually require a way to send and receive messages in an asynchronous way. Additionally, modern computing is all about events, and increasingly, integration solutions have to react to or push events.

Azure Service Bus Messaging provides reliable message queuing and publish-subscribe messaging capabilities. Azure Service Bus Queues and Topics allow us to decouple in time upstream and downstream systems or different interrelated services. Service Bus Queues provide ordered message delivery to competing consumers, while Service Bus Topics enable publish-subscribe messaging scenarios where multiple consumers can process the same message.

The following Service Bus features make it a key component of most integration solutions on Azure:

  • temporal decoupling
  • message processing load balancing (competing consumers)
  • Publish/Subscribe pattern via Service Bus Topics;

Azure Event Grid is a fully-managed event-routing platform on Azure which we can leverage as part of our integration solutions.  Logic Apps can react to events on Event Grid coming from the different publishers or publish events for other handlers to consume.

While Service Bus provides rich and robust messaging capabilities, like transactions, ordering, sessions, dead-lettering, enqueue time, long time to live and deduplication, among others; Event Grid offers hyper-scale event routing, with filtering, routing, built-in Azure publishers, with a push-push model, but a short time to live and no ordering. Additionally, due to these limitations, Event Grid is not meant for critical or transactional messages, but for events, which might still point to its source, e.g. the event of a blob being created containing the URL of the actual blob.

API Mediation and Management

API Mediation and Management is a core funcitonality of integration solutions which require to expose RESTful APIs.

Azure API Management is an Azure service which functions as a API gateway for backend APIs in the cloud or on-premises. Some of the benefits of API Management are:

Furthermore, Azure Functions Proxies provide a small subset of what API Management does, which can be leveraged as a light API Gateway for HTTP-triggered Logic Apps, including:

  • Azure Functions Proxies can be secured in a very similar way to App Services.
  • Modify requests and responses, which also allows us to mock APIs.
  • Consolidating multiple and disperse APIs into simpler URL routes.

Depending on the level of features we require, we can go with the lightweight Azure Functions Proxies or with the full API Management for our integration solutions.

Monitoring and Management

Azure Logic Apps provide built-in monitoring tools that allow you to check the run history, trigger history, status, performance, etc. You can also install the Logic Apps Management Solution on OMS, which gives you a very rich aggregated view and charts of all your logic apps that are being monitored. The OMS Logic Apps Management Solution view is shown in the figure below.

11 OMS Solution

Additionally, you can build custom activity monitoring dashboards and publish custom queries based on diagnostics sent to Azure Log Analytics. More information on my previous posts:

A sample of what you can build in the figure as follows.

12 BAM Dashboard

Development and Solution Life-cycle Management

To develop some of the iPaaS Solution components we can use the Azure Portal, however, when we think about the whole CI/CD pipeline, it’s better to work with Visual Studio and with Visual Studio Team Services (VSTS) for source control, build and release defections. More information about CI/CD for Logic Apps on my previous post:

Handy References

If you want to know more about Logic Apps, you might find the following links handy

 

Wrapping Up

In this post, I’ve discussed the current state of the Microsoft Azure Integration Platform as a Service (iPaaS) components, being Logic Apps the main player and complemented by other Azure product offerings. As mentioned previously, one of the main advantages of Microsoft in the iPaaS market is the whole Azure ecosystem and how we can build integration solutions leveraging the capabilities of all these different platforms. And we need to consider not only the technologies described here, but also all those which can be utilised through the connectors; such as Azure Cognitive Services and Azure Machine Learning.

Additionally, many of these components offer per-execution price or affordable entry-level pricing options. This enables us to start small when building enterprise-class integration solutions and grow from there.

I hope you’ve found this post useful, and please feel free to ask any questions or add your comments below.

Happy clouding!

Follow me on @pacodelacruz

Cross-posted on Deloitte Engineering Blog