About the construction of service-oriented architecture at enterprise
In my point of view one of the general, if not the main task while constructing the service-oriented architecture at enterprise is provision of the possibility of continuous development of the integrated system and its adapting to the changing business of customer-company business. And besides, ideally the development of the system must be implemented by the forces of customer IT-department with the minimal attraction of specialists building the system. I.e. There shouldn’t be so that for connection of new business system the customer-enterprise will need to invite consultants again which will begin from remodelling the overall integration of the existing systems, and then will fill something resulting in the new one. Or will not insert…
In order to resolve the given task it’s necessary to provide the implementation of the following conditions:
- The transition from the integration on the basis of point-to-point to the integration through the enterprise service bus (ESB).
- Ensuring the independence of the integration of transformed message formats.
- Documentation in sufficient quantity and quality.
- The presence of the monitoring process, transfer and process messages.
The transition to the integration through the enterprise service bus (ESB)
The systems integration on the basis of point-to-point has one significant drawback ― low scalability. The number of possible connections between systems can be calculated using the formula N(N-1)/2. If you integrate three systems, then the number of connections between them is also equal to three. In this case, the integration on the basis of point-to-point is justified. If there are 10 systems, the number of connections is equal to 45, and this is the fairly large value.
The problem is not just that here are a lot of connections and they can be confusing, but the fact that they all need to ensure that both the level of data sharing (single-unit can be used as a protocol – HTTP, the other – RMI, the third – JMS, and the N-th – Oracle AQ, while initially the source system at all except to put a message to the database table cannot do anything), and at the level of data format (single-receiver anticipates for the sharing file 1C as XML, other – IDOC, the third ― some customised certain XML, and N-th ― the SOAP message). Business applications should provide support for various protocols and data formats in the case of integration on the basis of point-to-point which is usually expensive and sometimes impossible in principle.
However, when there are few systems, for example three, everything is not enough clear. Now there are three, and probably it’s planned to connect the new system and not just one. With these customer plans it is desirable to acknowledge in advance.
Ensuring the independence of the integration of transformed message formats
It is possible to encapsulate the message transformation in ESB from the format of the source system to the format of the receiving system. Actually this is one of the main functions of ESB. The question is how to do it correctly.
Often in the literature devoted to ESB, you can find such a term as “canonical format”. Usually this term is understood as the maximum generic message format that allows you to send messages from any system to another regardless of the format in which the message came into the bus and of the message format for the recepient system. The development of both the canonical format and transformations from/to it is quite time-consuming task, so the desire to save money on these works is quite comprehensive. When this can be done?
First, if the main receivers of the messages are the systems of one manufacturer with the “native” exchange format (such as SAP IDOC), then as a canonical format you can use the “native” format. The transformation from this format will be carried out only in the adapters of the source systems .
Second, if the communications are the main sources of the same manufacturer, with its “native” format for the exchange, then as a canon, you can use this format. The transformation of the format will be carried out only in the adapter system of the recipient.
Third, if some systems are weakly or do not connected at all with the other, and connected only with each other, then maybe it makes no sense to do double transformation (first in canonical format, and then – in the format of the receiver system), especially if they are the systems of one manufacturer with a single exchange format. However, there is a need to keep in mind that if you will need to provide the interaction of these systems with any other tomorrow, you’ll have to develop a transformation, in some cases – very non-trivial.
Generally on the question, whether to use or not the canonical format you should proceed from the fact what is easier: to make the transformation to/from a certain intermediate format or directly from the original message to the required ones. The decision should be based on an analysis of the existing system landscape and the road map drawn up of the project. Knowledge of the customer plans of the further development of the system will also be useful. If it is expected the future replacement of the source systems, you will either have to lay the transformation of messages in data adapters systems or yet to introduce a canonical format.
Documentation in sufficient quantity and quality
Unfortunately, the amount of entropy in the universe is increasing. Over time, especially in a large system integration with many dozens of endpoints it will be very difficult to understand. Rising complexity will lead to that the developers involved in the creation of new or supporting the existing applications will develop their integration points and processes instead of using and adapting the existing ones. Actually this fact leads to a leveling role for SOA and to the fact that after some time to implement SOA should again have an updated view of the landscape.
To avoid the use of leveling embedded solutions needed in addition to technical means (the use of different registries and repositories), and provide a number of organizational measures such as compulsory and qualitative documentation of the landscape. The documentation should reflect the full information on all types of services (utilitnym, services, applications, business services and service processes):
- Name of the service;
- Integration style (batch, real-time);
- Calls style – how the call of service/application is provided (by the trigger, using the Change Data Capture, etc);
- Template exchange messaging (Request-Reply, One-Way, Pub-Sub);
- Business communications domain;
- Message format (XML, CSV, SOAP, IDOC, etc.);
- The exchange protocol (HTTP, MQ, RMI, etc.);
- Service contract (WSDL, WADL, etc.);
- Security (authentication, authorization, encryption, Non-repudiation);
- The level of security implementation (at the level of the communication channel, at the level of communication);
- Implementation of Security (WS-Security, SSL, etc.).
The schemes of the business processes that involve services should be separately reflected, the order of their calls and the rules of the interprating and received as a result of its work information.
But illusions are not worth it, the growth of entropy also applies to the documentation. In order to evolve the service-oriented architecture in harmony it is necessary to apply range of measures which should include both organizational and technical measures.
The presence of the monitoring process, transfer and process messages
Another of the advantages of implementing a service-oriented architecture, for example, in the form of ESB, is to provide a through monitoring of the transmission and processing of messages. Through monitoring enables to solve the following tasks:
- Provision of SLA;
- Acquisition and analysis of statistics regarding use of services;
- Control over the passage of each message over the bus;
- Control over the message processing in the business systems.
Provision of SLA
If you encounter problems in the integration solution, or at any other breach of SLA, such as a sharp decrease in the rate of integration bus, the pass-through monitoring subsystem must immediately notify the system administrators. System administrators should be able to obtain information about the performance of each component of the integration solution and react appropriately: fix the broken connection to the business system, switch to a backup link, to solve the performance problems with database and so on, depending on the situation.
Preparation and analysis of statistics regarding use of services
Maintaining the statistics of services use includes not only monitoring each of the call cases of service in order to detect unauthorized access or unexpected case of the use of business systems by company’s business processes. Statistics should be also maintained in order to optimize the loading of all enterprise information systems. For example, if the monitoring reveals that some service is not used, then you should find out the root cause: is it unavailable for any reason or was it gradually excluded from all company business processes, but wasn’t this fact noticed at once? It is possible that such service should be put out of operation with the relevant organizational and personnel decisions. Another option is to optimize the information infrastructure of the enterprise. For example, if the load on the service A is much lower than estimated, it is possible to transfer this service to the less expensive equipment, using the freed ones for other purposes.
Control over the passage of each message on the bus
Above all this monitoring is required to find the lost messages, as well as to find an answer to the question: why did the business-systems A and B contain the inconsistent data. Messaging transfer integration to bus proceeds in several stages, such as: registration, routing, delivery. If due to changing business needs there is produced adaptation of the bus, then typically this adaptation occurs on a “need to get it done yesterday”. When working in such an environment it is possible to forget something or make something wrong. In particular, not properly set up the routing, resulting in a jam message on the bus. To identify these problems and adequately correct them there must be a monitoring tool for the passage of each message through the bus. Report on the passage of messages on the bus must contain at least the following information:
- The message ID in the source system;
- The message type;
- The soruce system;
- The receiver system;
- The message ID in the system, the receiver (if any);
- Status reports (on which phase is in, or not transmitted to the receiving system, if processed);
- Information about the error (in case if while the transmission or processing an error happened).
Control over the message processing in the business systems
The correct message delivery into the business system is not the end of the performance of the business process. The message must still be processed in the system: parsed, the corresponding business objects should be created according to its data or other actions should be performed, the result should be returned (in case of an exchange on the basis of the request-response). At the same time there could be situations when the message can not be correctly processed for various reasons, such as incorrect format or absence of the needed data in the destination (in fact – the violation of referential integrity), and also errors in the business logic of the system. Apparently ensuring the proper handling of messages requires readjustment of the exchange. In any case, to determine the need for such a migration will not be possible if the business system will “swalowl” the cases of wrong message handling. To prevent this from happening – you need to configure the business system notification of the bus about the results of processing of incoming messages.
There are two possible variants depending on the type of exchange:
- Synchronous exchange. Since the exchange is simultaneous, i.e. the message is processed immediately after delivery to the receiving system and the result is formed, it is required to notify the bus about the results the message processing. The bus having received a result, must record the corresponding status of the message processing.
- Asynchronous. In this case it is possible that the result of message processing is not formed at all (the so-called one-way exchange). At the same time, as mentioned above, it is required to notify the bus on the results of message processing. You must provide another channel of communication between the business system and the bus where the notifications will pass through: was the message handled successfully or the message processed unsuccessfully. If an error occurs when processing the message it is necessary to transmit the text and the code of this error.
In the comments I suggest to express your views on the application integration, implementing SOA/ESB, or to ask any questions. Thank you for your attention.
P.S. Reserving the right to make changes to this article in order to increase relevance and consistency of the content.