Service orchestration

The services you identify, design and implement are dependent on the types of services you want, and how you structure them. Let us assume these layers:

• process orchestration – implementing business processes with both automated and manual steps.
• activity services – implementing fully automated processes within broader business processes. The activity services may use information from different information domains, like a payment process would use both services within the “customer” and “order” domain.
• information domain services – implementing operations on information entities, across the system portfolio, using integration services towards several systems and aggregating information.
• integration services – implementing operations on information entities towards a single system.

The service registry / repository established in a lot of IT organizations, enables reuse of already implemented services, but that alone tends to drive the actual reuse towards low-level services, like integration services towards specific systems, or quite specialized information aggregation services. The reason for this, is that process and process activity services in most cases are implemented based on a demand from an application development project, while the simpler services handling more basic information objects, implicitly match the needs for many projects to come. In order to get a higher potential for reuse of process / activity services, one needs to broaden the perspective to more application developments and, even more importantly, process improvements, at once.

The link between EA and SOA

EA is about planning ahead to reach a target architecture that captures the needs of the whole business, the needed capabilities of IT, and using that to scope and define the right projects, that iteratively will implement that target – through both business and IT initiatives.

SOA is about defining business-oriented functionality, which encapsulates what IT delivers to the business – as a whole. The most successful implementations of SOA is where you “think big, but implement small”, hence iterative implementation of the big picture.

The link between EA and SOA is obvious, especially when it comes the harmonization of needs across the business, and the holistic approach to IT.

The models within EA are the “tools” that enables the identification of the right services within a service oriented architecture. Examples of this are:

• a process map, and the process design of business critical processes, can be used to find the process and activity services that would create most value for the business, f.ex. by explicitly enabling most reuse
• conceptual and logical information models within the information architecture, should be the basis for a common object model in a SOA context, hence also  information domain / information entity services.
• in order to find the right consumers of services, and the right providers of system interfaces, used by integrations services, functional application architecture and system architecture are two important enterprise-wide architectural perspectives.

By identifying which different types of information entities that flow through the business processes, and establishing a common ground for important information objects, you have a very good starting point for defining information services that could be reused in those processes. F.ex. if a customer, seen from a business perspective, is either a private or a corporate customer, and these two types of customers have little in common, the service composition should reflect that, hence a service simply called “CustomerService” would probably be inadequate and very difficult to make reusable.

EA is an enabler for reusable process activity services

An organization must be dedicated to enterprise architecture work if it wants to take advantage of the possible EA/SOA synergy effects. Both the business units and the IT departments should think of enterprise architecture as a way of defining the goals the business and IT want to achieve and, more specifically, which processes they want to improve.

process map with value chains

• From the to-be process map, one needs to choose which value chains or which business processes from different value chains that are to be implemented. An insurance company may choose to improve the whole value chain for car insurance customers, or choose to improve the claim handling for all insurance types.
• Business processes that, from a conceptual perspective, have overlapping sub-process or process activity definitions should be identified, so that coherent processes could be improved together, either in a single project or through coordinated projects.
• Then these processes must be designed in more detail, from a business perspective, with the emphasis on how they want things to work, not the limitations of today.
• These processes must then be analyzed together, in what I call a process / activity matrix

• similar activities in different processes can now be identified, and these represent potentially common activities, hence reusable activity services. For a public organization managing a lot of different applications (building permits, kindergarten applications etc.), there are some decisions that could be automated, while others need to stay manual. The activity automation candidates here would imply good candidates for service reuse as well.
• some activities in the different processes may be doing some common “sub-activities”, but differ too much to constitute a potential reuse – in this case the activities should be modularized or consolidated, in order to find the common activities, hence the reusable services with the most adequate granularity. Within a manual decision process, the activities that publish or send the decisions may still be automated, hence reused from the automated decision activity.

process / activity matrix

Using tools to further enhance reuse potential

It is impossible to discuss such potentially complex intellectual tasks without mentioning tooling. If all business processes, their sub-processes, their automatic and manual activities, are modeled and designed in the same tool, and it is all well structured in a common repository, the steps mentioned above will become much smoother.

Consider this scenario: A business analyst in a bank models the loan application process, in which he defines an activity called “evaluate customer”. That activity is later linked to a service that implements this evaluation, which implies a relation to a credit check service as well. Somewhere else in the same bank, another business analyst models the customer registration process, where the need for screening the customers becomes apparent. Then he sees the “evaluate customer” activity in the repository of business processes, so he decides to put that activity in his own process. Then he has implicitly created a relation between his process and a couple of existing or planned activity services, which means that those services are going to be reused.

Enterprise architecture – planning – improvements – services

So, back to where we started: enabling service reuse through enterprise-wide architecture activities – can you achieve this in practice?

As I’ve already mentioned, it requires dedication from both business representatives and IT architects, and clear mandates to those who are in between: enterprise architects. If EA is used as the basis for project portfolio management, and the organization is serious about business process management and service oriented architecture, taking advantage of the link between EA and SOA is just the natural step for harvesting the actual benefits of being service oriented.

The Norwegian Smartgrid Centre works on a broad front to ensure that society has access to the necessary expertise in connection with the establishment of, and adaptation to, a new, improved energy supply system. The goal is a safe, efficient and environmentally friendly system providing better utilization of the electricity supply grid and increased use of renewable energy sources.

greenbird Solutions AS provides an advanced SOA based integration middleware platform for smart metering (rollout and operations) and smart grid solutions.

greenbird will bring in its experience and expert knowledge within smart integration solutions to contribute to the Smartgrid Centre’s activities such as research, teaching, business development and commercialisation, laboratory, testing and demonstration projects, network building or information.

For more info, please visit greenbird’s smart metering and smart grid focus area or take a look at the Norwegian Smartgrid Cntre’s webside [external link].

• collects information from core business applications (CIS, NIS, MDM, AM, etc.)
• profiles and analyzes data
• generates metrics
• monitors and reports data quality

greenbird’s Smart Metering Data Quality Platform helps DSOs, grid companies and utilities to overcome data quality issues and integration challenges with smart metering rollout and smart metering operations.

For more info regarding greenbird’s Smart Metering Data Quality Platform, please contact info[a]greenbird.com.

Introduction
greenbird is working with Europe’s leading Credit Management Services company, Intrum Justitia, to create a SOA enabled integration solution targeting the energy industry in Norway.
The solution expose a set of generic APIs plus a number of client specific adapters and integrates with a back end debt collection system by exchanging flat files. A large amount of integration points and data transformations are involved.

The solution is implemented on webMethods Integration Server from Software AG, a best of breed integration platform with a “point and click” based IDE called webMethods Developer.

Integration Server has great support for SOA enablement and application integration. It also has great support for manual testing, including a debugger with stepping and breakpoint capabilities.
It does however not come with built in support for automatic testing.

We are following an agile process and must be able to continuously handle new and changing requirements. Due to the “point and click” nature of the Integration Server IDE a lot of manual work is involved when refactoring existing features and adding new ones. It was therefore even more important than usual for us to write robust unit and integration tests to discover bugs and regressions as early as possible.

We wanted to use a testing framework to make the process of writing and maintaining high quality tests as effortless as possible. There are a few commercial frameworks that enables the writing of automatic tests directly in Integration Server but we wanted to leverage existing infrastructure and skill-sets based on Java and the JUnit framework. We found an Open Source JUnit based framework called wmUnit but the project has been inactive since 2006 and lacked some of the features we wanted. In the end we decided to write a simple framework ourselves.

These were the main requirements for our framework:

1. The tests must be able to run in a Continuous Integration set up.
2. We must be able to write the tests as regular JUnit tests.
3. The plumbing involved when doing remote calls to the Integration Server must be invisible.
4. It must be very easy to produce input data.
5. It must be very easy to perform assertions on the data returned by the Integration Server.

The requirement regarding Continuous Integration support was easily attained by fulfilling the requirement that the tests should be regular JUnit tests. There is a whole plethora of CI products supporting JUnit based projects.

The rest of this article gives an overview of how we implemented the other requirements.

JUnit based tests

We created a generic JUnit based superclass that makes the utilities described below easily available for test subclasses. We based the framework on the latest version of JUnit to ensure that the tests could be written using modern paradigms such as test annotations and Hamcrest matchers.

Transparent log-in and session management

The superclass takes care of logging into the right Integration Server and maintaining a logged on Context for the tests.

Transparent invocation and service selection

By using a convention over configuration approach the superclass is able to determine what Integration Server service is under test simply by analyzing the fully qualified name of the test class.
The convention is that the tests are placed in a package reflecting the folder structure of the service and named according to the following pattern: <service name>Test.
E.g.: To test the service my.folder.structure:myService you create a test with the following fully qualified name: my.folder.structure.MyServiceTest.
This enables tests to invoke the service under test with no configuration what so ever:

invoke(inData);

Cursor management

Integration Server returns data in IData structures. IData requires that you access data by opening a cursor. The cursor needs to be destroyed after use. The test superclass manages the top level cursor so that simple return values can be looked up without thinking about resource management:

IDataCursor responseCursor = invoke(inData);
String result = getString(responseCursor, "result");

IData builder DSL

The IData programming model is very verbose and requires resource management. Building even the simplest multi-level input documents is time consuming and error prone.
As an example we’re going to build an IData structure reflecting the following document model:

<rootElement>
        <subElement>string value</subElement>
        <subDocument>
                <name1>value 1</name1>
                <name2>value 2</name2>
         </subDocument>
</rootElement>

This would require the following IData code:

IDataCursor cursor = null;

IData subDocument = IDataFactory.create();
cursor = subDocument.getCursor();
IDataUtil.put(cursor, "name1", "value 1");
IDataUtil.put(cursor, "name2", "value 2");
cursor.destroy();

IData rootElement = IDataFactory.create();
cursor = rootElement.getCursor();
IDataUtil.put(cursor, "subElement", "string value");
IDataUtil.put(cursor, "subDocument", subDocument);
cursor.destroy();

IData document = IDataFactory.create();
cursor = document.getCursor();
IDataUtil.put(cursor, "rootElement", rootElement);
cursor.destroy();

We wanted to be able to easily define smaller documents directly in the test code so we created a simple Java based Domain Specific Language (DSL) that hides the complexities of the IData API. Here’s our example document model implemented using the DSL:

IData document = doc(
        el("rootElement", doc(
                 el("subElement", "string value")
                .el("subDocument", doc(
                         el("name1", "value 1")
                        .el("name2", "value 2")
                ))
        ))
);

The DSL enabled us to quickly define documents, elements and lists without worrying about the IData programming api and resource management.

Converting XML to IData

The DSL is suitable for smaller input documents, but when the document size get bigger it’s usually sensible to maintain the test data separately from the test code. Integration Server provides a couple of services for for converting XML to IData (pub.xml:xmlStringToXMLNode and pub.xml:xmlNodeToDocument). We leveraged these services to enable the tests to store larger input documents in an intuitive XML format. Our example document can simply be saved on the format described above:

<?xml version="1.0" encoding="UTF-8"?>
<rootElement>
        <subElement>string value</subElement>
        <subDocument>
                <name1>value 1</name1>
                <name2>value 2</name2>
         </subDocument>
</rootElement>

The input data structure can then be loaded into the test using the following one-liner:

IData inputData = toIData("my_test.xml");

Converting IData to XML for easy querying

Querying an IData document using the Integration Server Java API is just as verbose and error prone as building one. We decided we rather wanted to work with XML since there are many Java tools for querying XML documents.
Again we leveraged an existing Integration Server service (pub.xml:documentToXMLString) to convert incoming IData structures to XML documents.

We chose the XPath utility from the Java SDK as our query tool. We also created a little wrapper for it (XPathRoot) to make it simpler to use.

This made it easy for tests to query even the most complex IData document using xpath:

String xml = toXML(myData);
XPathRoot docRoot = xPath(xml);
String value2 = docRoot.value("/rootElement/subDocument/name2");

..or if you prefer a one-liner:

String value2 = xPath(toXML(myData)).value("/rootElement/subDocument/name2");

FTP support

It is very easy to set up an FTP endpoint on Integration Server so that services can be called with the content of files received from clients. Integration Server can also return the result of the service invocation as a file with a name on the pattern <input filename>.out. This makes FTP a viable transport for synchronous request/response style service invocations.

We created a simple way to test our synchronous FTP endpoints by integrating the Apache Commons Net FTP client into our test superclass. Files can now be transmitted and the response received by a one-liner:

String resultXml = ftpStore("file_name_to_put.txt", inputStream);