An Enterprise Guide on Service-Oriented Systems Integration

Service-oriented architecture and later microservices have become highly beneficial providing organizations with the level of flexibility they need. It is especially significant today when the number of applications adopted by companies grows each year, demanding more efficient governance approaches.

Number of Applications per Company

Source

What Is SOA and Its Role in Software Design?

As Red Hat defines, “service-oriented architecture (SOA) is a type of software design that makes software components reusable employing service interfaces based on a common communication language over a network”. With 35% of organizations choosing service-oriented architecture for their system design, it has become a proven vendor- and technology-agnostic approach to integrate autonomously deployed and maintained software components. SOA enables efficient communication between services by passing data or by coordinating the activity of two or more services.

Each service in the SOA performs as an independent piece of software with its responsibilities, user interface, databases, supported protocol, etc. A good example of a SOA-based system is a set of customer services, like CRM, ERP, Product Information Management System (PIM), etc. These services can be implemented using different technologies and support diverse protocols of communication, data models, etc. However, to enable effective service-to-service communication and align their workflows, the system needs an additional integration mechanism in between.

Most Common Service Integration Approaches

Integration is a backbone of SOA-based system and can be implemented in different ways depending on business demands, platforms, and SOA best practices.

Here we will take a closer look at some of the most common SOA integration approaches.

If you are looking at how to gather all your IT systems, services, and applications within a unified IT landscape, have a look at Infopulse’s advanced integration operations services to sync your data and optimize operations across different sites.

Direct Connection

In architectures, powered by direct connection, services can communicate with one another using HTTP requests. Sometimes services build a chain of requests in which the final response depends on the responses of all the services involved in the chain.

Direct Connection Sample

Benefits: It is the easiest and fastest way of implementing connections between services. Most modern services have advanced API layers, mainly RESTful API endpoints. Using HTTP protocol makes it easy to connect services that are built and being hosted on different and incompatible platforms.

Drawbacks: Since the services are tightly coupled and rely on synchronous communication, they become completely dependent on one another and even a temporary unavailability of one service can block all other services.

Message-Driven Design Pattern

To address the challenges of tight coupling, you can opt for the Message-Driven Design Pattern that allows applications, services, and systems (producers) to send messages to a delivery service rather than to other services directly. Then, the delivery service brings messages to consumers (services that consume messages sent by applications).

The most common types of delivery services are Message Broker and Enterprise Service Bus.

Benefits: There are no direct connections and request chains between services and thus, the entire system works in an asynchronous way. With loosely coupled systems, each service acts and scales independently and if one service fails, other components of the system can still function. Together with high scalability and resilience, you can also overcome integration complexities.

Message Delivery Services within SOA

Message Broker Integration

A message broker is a lightweight service that simply delivers messages from producers to consumers without any possibility to extend its business logic. This middleware service allows services, applications, and systems to communicate freely with each other literally translating messages of different protocols. So, whatever languages the platform or apps use, they will still be able to “talk” with each other.

Benefits: A message broker is a reliable message delivery service that can support hybrid-cloud, cloud-native, serverless, and microservices architectures. It serves as a single endpoint for all the providers and consumers and allows for flexible communication between apps, services, and systems. Consumers can either pull the messages from the message broker or subscribe to topics and react to the message appearance events of the message broker. This results in higher information availability.

Message Broker Architecture

Challenges: To work with a message broker directly, you should significantly rewrite the code of services. This would facilitate the implementation of specific message-driven business logic and protocols. Especially if the Direct Connection integration has already been enabled. Another drawback is that changing business logic in one service may require rewriting the code in other services too. Moreover, a development team must have high expertise in architecture, infrastructure, and other aspects of a particular message broker. Besides, it is essential to ensure that platforms where services root in have rich facilities of interactions with the broker, e.g., SDK.

On top of that, enabling integration with the help of a message broker assumes that producers and receivers are responsible for establishing connection with the message broker and can only interact with the broker using its specific protocol. The business logic of the integration also rests solely on the producers that must know queues or topics to send messages to. While consumers must use the same queues and topics also taking into account the format of the messages. Besides that, the business logic of integration, specifically processing messages that cannot be delivered, should be implemented on the app level.

What Is Enterprise Service Bus?

Another commonplace and better option for services integration and handling information exchange within service-oriented architectures is Enterprise Service Bus. Primarily functioning as a centralized software component, it allows integrating apps and implementing necessary transformations through a service interface that can be further reused by new apps. ESB is also responsible for enabling connectivity, communication protocols conversion, transforming data models, routing messages, and handling composition of numerous requests.

Benefits: While a message broker is just a transport layer of a distributed app, an ESB is a comprehensive integration solution, which in turn includes a message broker and other services that cover all the integration aspects. Enterprise Service Bus helps set up integration flows, validate inbound data models, and transform data formats and models. It is also in charge for routing messages, guaranteeing their delivery, and handling any delivery failures.

Challenges: ESB is a complex solution with a lot of cloud services involved. This may complicate the infrastructure of the whole application and demand extra monitoring efforts.

Enterprise Service Bus Architecture

On-premises vs. Cloud-based Enterprise Service Bus Architecture

There are two ways of Enterprise Service Bus implementation ­– on-premises or in the cloud.

When on-premises, a customer has full control over the ESB infrastructure that is more favorable if you should meet strict security requirements to the application. However, in this case, the customer takes full responsibility for the infrastructure setup and support. The latter is a complex enterprise that suggests broad expertise and a lot of efforts of in-house developers to maintain highly available and scalable services.

Besides that, you will need to purchase the necessary equipment to match availability and scalability requirements during the peak-load periods. The support efforts should be taken independently on a real load of the system.

To address these on-premises limitations, many lean toward cloud-based ESB solutions that continue functioning without degrading in the environments with unpredictable throughput and load.

A cloud provider shoulders responsibility for the infrastructure creation and support, automatic scalability, failover, etc. The provider also guarantees a particular level of quality, availability, and reliability of their services to the customer. Plus, the customer does not need to purchase any equipment and pays only for the consumed resources.

Case study

IPCO Achieves Ultimate Flexibility with a Cloud-Based Integration Platform

Read the full story

The architecture itself is very flexible: ESB consists of logical blocks with distinct functionalities. Each block can be implemented by using different services or a combination of them. The choice of services depends on the Enterprise Service Bus requirements to the throughput, availability (desired SLA – service level agreement), the price of the system, etc.

ESB Architectural Components

Backend systems – customer’s services involved in the integration. The same service can send messages to the ESB as well as receive them from the ESB.

• Producers – services that send messages to the ESB
• Consumers – services that receive messages from the ESB.

Adapters – ESB components, responsible for interaction with backend systems. Adapters hide details of processes in the ESB and provide backend systems with a unified way to interact with the ESB, independently of the services platforms and technology they ground on.

• Inbound adapters – RESTful API endpoints represent inbound ESB interface and support HTTP POST method. Inbound adapters perform initial validation of inbound requests data models, transform messages to a compatible format of an internal message broker (if necessary) and send messages to message queues or publish them to message topics.
• Outbound adapters – represent outbound ESB interface. After taking a message from a message broker queue, the adapter transforms a message format (if necessary) and sends the message in the Request Body to a desired backend system (Consumer) using a POST method. Outbound adapters can also support a poll model allowing backend systems to request messages from the ESB themselves.

Transport – a message broker that accepts messages, buffering, and delivering them to the various components of the ESB.

Transformations – ESB components that transform the message format/model during the transportation and redirect them within the business logic workflow.

How Does ESB Work?

Customers’ producers interact with the ESB through RESTful API endpoints (Inbound adapters) and connect with consumers through the Direct Connection integration. Each Inbound adapter represents the beginning of a predefined business logic pipeline. Business logic can include inbound data models validation, different message data transformations, enabling message persistence until they are delivered. It also enacts the redirection of messages to one or many outbound adapter(s), getting them to the end of a business logic pipeline. Outbound adapters are responsible for delivering messages to corresponding RESTful, SOAP consumers.

A Typical Sequence of Tasks Integration in the ESB

A pair of inbound/outbound adapters are placed at the beginning and the end of an integration flow to ensure the delivery of a message from one backend system (Producer) to one or more backend systems(s) (Consumers). If, after a defined number of attempts, the outbound adapter fails to deliver the message to the Consumer(s) due to the unavailability of the Consumer service(s), the message goes to the Dead-messages queue for further analysis. In case the message is not delivered because of the backend system’s Bad Request response, the message goes to the Poison Queue for further analysis.

Enterprise Service Bus Use Cases

The main ESB functionality lays the ground to a range of use cases for SOA implementation. Below we will also mention some of enterprise service bus best practices.

1. Routing and messaging/data exchange (a set of Adapters, Message Broker, Transformation)

The routing components in the ESB environment enable:

• Connection of services to the transport environment
• Data transformation and routing
• Exchange of messages
• Information update
• Subscription to send messages
• Organization of asynchronous information transfer
• FI-FO message order (if required)
• Dead- and Poison-messages handling

The routing components use special dedicated software tools, adapters, and brokers (gateways) that provide connection to the integration service, transportation (data / message transmission) with guaranteed delivery, data transformation, and processing.

2. Enterprise Service Bus security and access control

The protection and governance components at the ESB level must provide:

• Protection and control of access to the services
• Encryption of transmitted data
• Connection protection
• Prevention of unauthorized access to information

Built-in authentication and authorization tools, as well as automation software tools, are needed to track and block unauthorized actions from integrated services and systems to prevent the leakage of sensitive data. Security measures must provide segregated access to the ESB facilities, industrial level of endpoints protection against DDoS and other malicious attacks, enable OWASP compliance, etc.

3. Monitoring, control, diagnostics, and tracking of integration processes (logging in)

The mechanisms of the monitoring, diagnostics, and tracking of the state of processes execution must provide:

• Identification of problems related to both data transmission and the status of operations performed on the service bus as a whole.
• Identification of potential problems in the initial stages before the problem manifests itself in full.
• Timely implementation of a set of preventive measures.
• Monitor and control of key indicators and metrics of business processes at different stages.

4. Guaranteed delivery of messages

The ESB must have facilities of storing messages until they are delivered to the addressees, buffering the messages in case consumers are not able to process them in real time during the peak-load periods. A reserve channel for messages delivery should also be in place in case the main one fails.

• Failover message broker cluster
• Autoscale of all ESB components
• Implementation of Retry and Circuit Breaker patterns in the message delivery to ensure high resilience

5. A single environment for ESB integration services

Gain from a unified user interface for integration scenario development, centralized management, control, monitoring, and administration of the ESB integration scheme.

For this, a user interface should contain a set of tools:

• Configuring adapters, transforming messages, queuing, and routing messages and data
• Management and configuration of data transport between ESB services
• Management and administration of control, monitoring and diagnostic services

Benefits: Enterprise Service Bus technology simplifies a message-driven integration of diverse services from heterogeneous environments. The backend system developer teams can focus on the backend business logic relying on the existing ESB integration facilities, which otherwise, should also be implemented. Thanks to the cloud and opened architecture, ESB within SOA-based systems can bring the highest resilience and scalability of integration and help optimize cost spending since you only pay for the consumed resources.

Conclusion

Comparing the benefits and drawbacks of both Message Broker and Enterprise Service Bus, we can assume that the most effective integration approach for enterprise-level SOA systems is cloud-based Enterprise Service Bus. Not only does it optimize development and support processes, but also brings in flexible pricing, high availability, reliability, efficiency, and predictability of costs.