Method and software apparatus for integration of business rule engine into data flow parallel processing frameworks (05-Mar-2009)

Method and software apparatus for integration of business rule engine into data flow parallel processing frameworks

Introduction

In the past, customers thought that such integration was not possible, or too complicated. That's simply because they didn't have the vision to come up with an elegant and generic enough integration solution. In one case, a customer simply abandoned both IBM InfoSphere DataStage and ILog JRules altogether, resorting to database stored procedures instead (in this case, there was parallel framework and no rules engine). In other instances, rules engines are used, but without a parallel framework - in these cases, there is a lot of custom coding involved for interfacing with data files and databases, as well as performing tasks that are otherwise supported out of the box by an ETL tool (like IBM InfoSphere DataStage).

There are APIs and methods that allow the integration of products like ILog JRules and IBM InfoSphere DataStage. This could be done with via WebServices (e.g., IBM InfoSphere Information Services Director), JNI, named pipes, etc. The different pieces (ILog JRules, IBM InfoSphere DataStage and integration APIs) are not new. However, the value of this invention is that it puts it all together and provides a generic packaged solution that hides the complexities of custom coding.

This invention is a generic solution for the integration of rules execution into a parallel DataStage job.

Invention Description

The invention consists of a set of aspects that provide a unique overall solution as follows:

Integration Architecture

The goal is to take advantage of the ultimate scalability provided by DataStage EE to process business rules. Business rule processing is not restricted to a single-threaded environment, taking advantage of just a single CPU. Instead, it may spread across multiple CPUs on a single SMP system, or across multiple CPUs of independent machines on an cluster, MPP or grid environment. DataStage EE takes care of data partitioning and parallelism. It manages the lifecycle of the execution environment, allocates resources, starts processes and establishes communication channels (data sets ) between them. Processing tasks such as sorting and read to/writing from flat files or database tables can be done by IBM InfoSphere DataStage stages, rather than by means of custom code to interface ILog JRules with external systems. Rules may focus on complex business requirements, instead of dealing with the aspects listed above.

1

Rule Evaluation Cycle

Rows flow through components of a IBM InfoSphere DataStage job and reach the input links of ILog JRules stage instances. The stage fetches and maps a subset at a time of the incoming rows to Java object instances. Those objects are asserted into ILog's memory and business rules are fired. Once rules are evaluated, the resulting objects are mapped back to IBM InfoSphere DataStage rows, which are wri...