A dynamic estimation method of the end times of batch processes on batch multi-processing servers and a system implementing the method. (21-Dec-2009)
Disclosed is a method and a system that can estimate as accurate as possible when batch jobs will end on a multiple processing system. Especially, they are valuable if the batch jobs are precondition for some online service. While web systems and 24 hours online service has become popular, batch data processing is still required to prepare data for online reference. In such systems, the performance and operational design of batch jobs to end before the desired start time of the online service might have been performed. Even in well-designed systems, however, some batch job may end abnormally or some interface file may not arrive at the system before the desired time. In those cases, operators need an accurate estimation of the end time of batch jobs which are precondition to some online service to perform some emergency action or to announce the estimated online start time to users. But the accurate estimation is too difficult to be done in a real environment. The difficulty comes from the different data volume processed and the different congestion degree on the system day by day and time by time. If some trouble happens in a current system, the operator or the administrator recovers it at first, and then he/she guesses empirically the processing time of the successive series of batch jobs that are preconditions for the online service. But it must be a rough estimation. If the online service will start much later than the estimated time, the users of the system will make more complaints. If it will start much earlier than the estimated time, they will not believe estimations any more. If this accurate estimation system can be used, the operators can stop batch jobs having lower priorities or raise the priority of jobs on the critical path. Moreover they can announce more accurate start time of the online service to their users, and it will raise the customers’ satisfaction because they can do another works with ease.
A dynamic estimation method of the end times of batch processes on batch multi-processing servers and a system implementing the method .
1. Definition of terms and applicable system environments
Most of business systems are realized with a combination of batch processing jobs and
online transaction systems on multiple servers. The system environments that this disclosed method and system is applicable to are those consist of multiple servers on which multiple batch application jobs are running with a job scheduler product. But in the following sections, the case consists of one server is described for readers easy to understand.
The definition of terms in this disclosure is described in the followings.
Batch processes running on a server usually consist of small units that are scheduled and execute. This small unit is called as a "job".
A
job consists of a conditional sequential execution of multiple programs, and the smallest executable unit is called as a "job-step".
A
job may have preceding and subsequent dependent jobs, and it is registered in and managed by a job scheduler product. In order to make the registration and administration of
j
obs easy, a group of jobs called an "application" is defined on some scheduler product. But the "application" concept is not essential for this disclosure and is omitted. Each job can be dependent on not only other jobs' normal ends but also a start time condition, external file arrival conditions, or manual release conditions of preceding dependent jobs by an operator or an online user.
In the job scheduler product, multiple "job scheduling patterns" that have different operational requirements depend on schedule dates, for example, a working day, a holiday, a billing cut-off day, a specific day of a week, a start day of a month, etc, which is defined with each job's "run cycle" and a system-wide "calendar". Therefore jobs scheduled to execute and their dependency on some day, which are called as a "job network" of the day, are usually different from those on the others.
The data amount processed by each job may vary day by day, so each job's elapsed time could vary largely depend on the data amount processed, or executing jobs simultaneously, or some wait time for some special resource. Those elements are usually difficult to be anticipated.
group of jobs that performs same kind of business work is called a "subsystem" and each
job may have a different "execution priority".
2. Workload indexes and standalone job-step elapsed times
t first, considering the indexes that indicate the workload of jobs in a subsystem, choose representative values of subsystem workload, for example, record counts of daily transactions, those of master DBs, and accumulated record counts for monthly batch processes etc, which are called "workload indexes". Each index is represented as Ln (n = 1, 2 …) in the following description.
These workload indexes are set with estimat...
Find More Documents Like This
Download this Disclosure
Click here if you want to sign-in now.
