| |
| |
|
|
Platforms
Job Scheduler is operated for the following platforms:
-
Windows 2000/2003/XP/Vista
-
Linux starting with kernel 2.4
-
Solaris 8/9/10
-
HP-UX 11 (PA-RISC, IA64 Itanium)
Job Scheduler can be operated without database or with one of the supported databases
by ODBC/JDBC drivers:
- Oracle 8.1.7, 9.2, 10g
- SQL Server 2000, 2005
- DB2 8.x
- MySQL version 4.1, 5.x
- PostgreSQL 8.x
- Firebird 1.5
|
|
|
Organization of Jobs and Job Chains
The Job Scheduler is used in order to launch executable files and shell scripts and to run database procedures
at a predefined point in time or triggered by external events.
Job Execution:
-
Jobs are the basic unit for the processing of executable files, shell scripts, procedures
and of job implementations based on the Job Scheduler API.
-
Jobs can be executed independently from one another. However, depending on the
execution result (success, failure, exit code) of a job a successor job can be started.
-
Jobs can be executed in parallel to a configurable number of tasks.
Read more on this feature in the job documentation.
Job Chains:
-
Job Chains can be seen as an assembly line on which multiple job nodes are passed.
Therefore, each job comprises exactly one step in the processing of a chain.
-
The workflow is regulated by orders. An order can be thought of as a directive which is processed in a
chain of jobs. An order is assigned to a job chain, with an identifier which is valid within that job chain.
The order also has a status which changes after the processing of each job node and can have a payload of parameters.
-
Orders are persistently stored during processing. This means that if a job is stopped during processing
and then restarted, it will be continued at exactly the point where it was stopped.
Read more on this feature in the job chain documentation.
Organization:
-
Job and job chain configurations can be read from configuration files of a host.
-
Configurations can be read from a central database on startup of the Job Scheduler.
This concept is dealt with in the topic Managed Jobs.
-
Configurations can be deployed to Job Scheduler instances
on multiple hosts by use of Managed Jobs.
|
|
|
Standard Jobs
The Job Scheduler is distributed with a set of standard jobs:
-
Logging and Cleanup
-
Sanity Checking
-
Mail Forwarding
-
Remote Job Execution
-
FTP Transfer
-
File Operations (rename, copy, remove, check existence)
Read more on this feature in the solutions section of the web site.
|
|
|
Solution Stacks
Solution Stacks are implementations of the Job Scheduler with third party components.
They might include code that is not compatible with the Job Scheduler's open source license,
therefore they are not part of the distribution, but available by separate download:
-
Integration with Network Monitors (Nagios, Hobbit)
-
Reporting with JasperReports
-
Secure Shell with Ganymed (ssh, scp, sftp)
Read more on this feature in the solutions section of the web site.
|
|
|
Configuration with Hot Folders
Hot Folders are directories that are automatically monitored by the Job Scheduler.
Configuations for the Job Scheduler can be maintained in seperate files, in case of
any change the Job Scheduler takes over the new configuration during runtime with no necessity for a
restarting.
Read more on this feature in the Configurations with Hot Folders.
|
|
|
Graphical User Interfaces (GUI)
The Job Scheduler is configured by XML files. However, you can use the GUI
to manage jobs without any knowledge of XML. GUIs are available to
control jobs at run time and to manage jobs and job chains.
Built-in Job Control GUI
Each Job Scheduler contains a HTTP server which shows job status information in a GUI
and allows to control jobs (start and stop jobs, access log files etc.) at run time:
-
Shows the job history and protocols for tasks, orders and the main Job Scheduler logs.
-
Provides functions to monitor and control the job scheduling: start jobs, stop jobs, terminate, set parameters, call up protocols.
Read more on this feature in the HTTP server documentation.
Managed Jobs GUI
It is possible to control and monitor several Job Schedulers on different operating systems using one GUI.
-
An explorer-like web interface based on an Ajax implementation is used to configure jobs, job chains
and all sort of Job Scheduler objects.
-
Jobs can be submitted to multiple Job Schedulers on different hosts.
-
Job Scheduler configurations are stored in one of the supported databases.
-
The web interface provides access to the job history and
enables you to retrieve log files of previous job executions.
Read more on this feature in the Documentation Managed Jobs (PDF).
Graphical Editor for XML Configuration Files
The graphical editor is used as an alternative to the Managed Jobs GUI
for users that do not want to store job configurations in a database,
but prefer to store them in files on disk.
-
The grafical editor reduces the complexity of directly accessing the XML structure
and validates the configuration against an XML schema.
Moreover, it is far easier to manage complex run time job configurations by mouse click.
-
The grafical editor supports the same operations as the Managed Jobs GUI.
-
The editor provides wizards for the configuration of jobs and job chains.
Read more on this feature in the configuration editor documentation.
|
|
|
Command Line Operation
Besides the Graphical User Interfaces the Job Scheduler can be controlled
by a command line utility for basic administrative tasks:
- Start, stop, terminate and cancel the Job Scheduler
- Start, stop, terminate and cancel jobs.
Read more on this feature in the command line documentation.
|
|
|
Application Programming Interface
Job Scheduler provides APIs for the implementation of jobs, job scripts
and for remote processing control.
API for Job Implementation
-
Script jobs in any of the supported languages Java, Javascript (Spidermonkey), Perl, VBScript.
You could use any scripting language that you like, however, job implementations in the
supported languages have access to the Job Scheduler API.
-
Add scripts for pre- and post-processing of existing jobs (monitor scripts).
-
Start jobs and job chains programmatically.
-
Create dynamic job parameter sets.
-
Implement conditional job processing.
-
Write info messages and debug messages to the Job Scheduler log file.
-
Send e-mail from your jobs.
-
Create dynamic job configurations, add jobs, job chains and orders programmatically.
Read more on this feature in the API documentation.
XML API for Job Control
-
Control jobs and job chains by simple XML commands.
-
Start jobs, add and remove orders for job chains.
-
Control the Job Scheduler (terminate, abort, restart, pause, continue).
-
Retrieve status information from the Job Scheduler on jobs, job chains, orders etc.
Read more on this feature in the documentation of XML commands and responses.
|
|
|
Web Service Integration
The Job Scheduler can be configured to behave as a Web Service for your jobs and job chains:
-
Any job can be exposed as a Web Service - independent from its implementation.
-
Handling of Web Service requests includes synchronous/ asynchronous job starts.
-
Requests of Web Service clients make use of the Job Scheduler XML API for Job Control
in order to start jobs and job chains, and to pass parameter sets.
-
Requests of Web Service clients can be automatically transformed by stylesheets to the Job Scheduler XML API.
-
Responses the to caller or to other Web Services are handled by the Job Scheduler.
-
Responses can be automatically transformed by stylesheets to specific XML formats.
Read more on this feature in the Web Service Implementation Tutorial.
See more samples for this feature in the solution section of this web site.
|
|
|
Job Start Events
Jobs can be started by the following events:
-
Job starts can be triggered by directory monitoring - should changes in directories, or the addition
or deletion of files in directories occur.
Read more on this feature in the documentation
directory monitoring
and file orders.
- Jobs can be started when a pre-defined point in time - time of day,
weekday, day of the months, relative to end of month (ultimo) etc. - is reached.
Read more on this feature in the documentation of run times.
-
Program-controlled job starts can be made using the Job Scheduler API
(available for Java, Javascript, VBScript, Perl).
Read more on this feature in the API documentation.
-
Event driven job starts can be initiated by notifications that are sent via TCP and UDP.
Read more on this feature in the documentation of XML commands.
-
Manual job starts can be made using the graphical user interface.
-
Jobs can be started manually from the command line.
|
|
|
Execution Timeslots
Job activities can be limited to timeslots.
The Job Scheduler supports any number of timeslots,
which can be configured according to individual job requirements.
-
Planned system outages or system backup intervals can be configured that prevent
any jobs from being executed.
-
Timeslots are the allowed operating periods for jobs, e.g. between 8:00am and 10:00am,
that can be configured independently from the event that triggers the
start of a job, e.g. at 8:30am.
-
Job starts that are triggered outside of
a timeslot will be auotmatically postponed for the next available timeslot.
-
Exclusive timeslots for holidays can be configured per job or for all jobs.
Read more on this feature in the documentation on run time configuration.
|
|
|
Job Prioritization
The Job Scheduler allows the assignment of job priorities:
-
By assigning one of system priorities idle, below_normal, normal, above_normal and high
or the numerical values allowed for the operating system being used.
-
Using process classes, jobs are started as separate processes (tasks). It is possible to configure
a maximum number of processes per job. For example, if three jobs are configured for one process class with a maximum of
10 processes per job, then it is possible that one job is processed in multiple instances (tasks).
Read more on this feature in the job documentation.
|
|
|
Notification by E-Mail
Notifications about the status of a job can be automatically sent by e-mail.
E-mails can be sent automatically in the following events:
- on success:
In the case of an executable file, when an exit code = 0 is triggered; for a programmed job this event is triggered should no error occur.
- on error:
Should an exit code != 0 be triggered by an executable file or should an exception occur or should an API call in a job raise an error.
- on warning:
For programmed jobs this event can be triggered by a warning; warnings are created by respective API calls to the task log.
- on process:
This event can be triggered when a predefined number of processing steps have been carried out.
Content of automatically generated e-mails:
- Return address information about the host, the Job Scheduler and the time at which the error or other event occurred.
- The original error or other message.
- The job protocol as an attachment: for executable files
stdout and stderr are included in the protocol; for programmed jobs all logged output is included. Log levels for protocols can be configured.
- E-mails are sent to pre-determined recipients who can be individually determined for each job. Note that all the elements of an e-mail (recipient, Cc, Bcc, subject, body, additional attachments) can be configured using the Job Scheduler API.
- The content and layout of e-mails can be adjusted by individual stylesheets.
- If an e-mail can not be sent immediately, for example due to failures in the mail server connection, then the mail is stored in the file system and the Job Scheduler repeats the mail transfer at regular intervals.
Read more on this feature in the documentation on sending e-mails.
|
|
|
Job History Protocols
The following protocols are delivered:
- Job Protocol:
The start and stop of a job.
- Job Scheduler Protocol:
A summary of all the protocols of all tasks.
- Task Protocol:
A task protocol is created for every task of every job: for executable files
the protocol contains the content of stdout and stderr;
for programmed jobs the protocol contains the logged output, log levels can be
defined for each job individually.
- Order Protocol:
The order protocol shows information related to the completed tasks
of an order - in other words, it shows a summary of all processes an order runs through.
- Debug Protocol:
For debugging purposes.
With the exception of the analysis protocol, all protocols are automatically packed (gzip) and stored
in the database. The protocols can be retrieved either using the Job Scheduler graphical user interfaces
or the central job history user interface. This interface offers information about processing, frequency
and the duration of job workflows.
Apart from the log files, task and order history are stored in the database for additional reporting.
Read more on this feature in the documentation of protocols.
|
|
|
Job Locks
The locking feature prevents that two jobs access the same resource, e.g. a database, at the same time.
In other words, only one process at a time can receive the exclusive right to access the resource as
long as the lock is active. Locking safeguards for example that read and write requirements of files
and databases are consistent.
Should a job be waiting for a lock to be released (lock contention), then it will be automatically
started as soon as the lock has been freed
The Job Scheduler supports two types of locking:
- Exclusive Locks:
As long as an exclusive lock is assigned to a task (job), no other task can receive this status
at the same time. The exclusive lock makes sure that a process is not interrupted by a third
process while writing or reading into a resource (write-lock). The exclusive lock is used for
processes that are changing the resource.
- Non-exclusive Locks:
They can be assigned to several tasks (jobs) at the same time. The non-exclusive lock permits that
several processes at the same time access a resource, provided that the processes do not want
to change the resource. (read-lock) The number of non-exclusive locks can be limited.
Read more on this feature in the documentation of job locks.
|
|
|
Remote Execution
Should the need occur to have
one job executing on host A that should be continued by another job
that is executing on host B, then remote execution will be appropriate.
Jobs may be processed by a Job Scheduler running on a remote computer.
This is true for simple shell scripts and executable files that make use of the
above standard jobs.
-
Remotely processed jobs behave, with respect to the Job Scheduler which is processing them,
just the same as locally processed jobs. The only difference is that the processing load is transferred
from the initiating to the processing Job Scheduler.
-
This means, for example, that all API calls refer to the local Job Scheduler object.
-
The job log information, its end state and any possible error messages will be forwarded to the initiating Job Scheduler.
Read more on this feature in the documentation for remote execution.
|
|
|
Backup Job Schedulers: Availability and Failover
For enterprise level computing you have to rely on your jobs being executed
in spite of the fact that a specific Job Scheduler server system is down or has crashed.
A Job Scheduler backup cluster ensures fail-safe operation of a (primary) Job Scheduler.
The cluster comprises this primary Job Scheduler and one or more reserve (backup) Job Schedulers.
A fail-safe system consists of a primary Job Scheduler and at least one backup, with both these
Job Schedulers running on different computers.
-
All the Job Schedulers in a backup cluster show their own availability by sending out "heartbeats" and,
at the same time, checking whether the other Job Schedulers in the cluster are available
by monitoring their "heartbeats".
-
Should one of the backup Job Schedulers determine the absence of the heartbeat from the
primary Job Scheduler over a longer period of time (ca. 1-2 minutes), then it will take over processing.
This means that it will continue to process the open orders and jobs started by the primary Job Scheduler and,
if required, start new jobs.
Read more on this feature in the documentation of backup Job Schedulers.
|
|
|
Load Balancing with Multiple Job Schedulers
Should you have a high volume of data with time consuming processing, then multiple Job Schedulers
will speed up the processing time considerably and provide higher availability.
In load balancing mode
the processing tasks are shared between multiple Job Schedulers
that are processing distribute orders on more than one host.
-
Multiple Job Schedulers (cluster) can process orders from different hosts at the same time.
A Job Scheduler cluster is used in order to reduce the overall processing
time and to minimize the necessity for the implementation of additional backup hardware.
-
In a cluster Job Schedulers signal their availability by sending out heartbeats.
If one Job Scheduler instance fails then the task will immediately be taken over by one of
the other Job Schedulers.
-
An arbitrary number of Job Schedulers can be operated in parallel.
Read more on this feature in the documentation of load balancing with distributed orders.
|
|
|
Measures for System Robustness
Job Scheduler provides the following measures for resource shortage/ system outages:
- Sanity Checking
Each Job Scheduler provides a job for sanity checking at configurable depth.
Sanity checking can be carried out for the
the resources available for process starts, the available memory and disk space.
One or more Job Schedulers can monitor one another.
- Restart Capability
The Job Scheduler persistently stores information in the database about open
orders or enqueued jobs. After a restart the Job Scheduler continues with the
processing of orders and jobs where it left off.
- Restore Database Connections
The database connection is the single point of failure - if the connection fails,
a configurable procedure is activated:
- Repeatedly try to establish a connection,
- Abort connection retries after a specified number of attempts,
- Continue operation without a database. This is possible if no Managed Jobs are used or jobs that otherwise depend on a database. In this situation the job history will be maintained in local text files.
Should a database connection need to be restored then a notification
will be sent to the administrator.
Read more on this feature in the database settings documentation.
|
|
|
Execution of Database Procedures
The Job Scheduler can be used to execute jobs that include SQL statements, calls to stored procedures etc.
in one of the supported databases:
-
It is possible to execute an unlimited number of statements and procedures.
Return codes and SQL output are recorded
in the job protocol.
-
With the Managed Jobs web interface database
connection credentials can be configured centrally and stored non-redundantly.
-
Database statements and procedures can be started by Job Scheduler instances
on any platform. As the connections are established by Type 4 JDBC drivers
over the network the Job Scheduler does not have to be installed
on the database server. Moreover, the processing load of such jobs
is mainly in the database.
Read more on this feature in the Documentation Managed Jobs (PDF).
|
|
|
MySQL Job Scheduling
SQL Interface:
This interface can be used in order to:
- Run any SQL statements in batch mode (similar to Oracle dbms_jobs)
- Launch Stored Procedures at configurable timestamps
- Automate job repetition at configurable intervals
- Notify administrators by mail in the event of error or success
- Set permissions for users starting jobs
- Grant individual permissions for job submission
Read more on this feature in the MySQL Job Scheduling documentation (PDF).
Maintenance Monitoring:
- Refresh database statistics (analyze table)
- Check database consistence (repair table)
- Monitor replication instances: errors in MySQL replication are instantly reported and administrators notified by mail
Read more on this feature in the MySQL job solutions web site.
|
|
|
Job Scheduler Development Roadmap
The PowerPoint presentation
and PDF presentation
show the features planned for the Open Source Job Scheduler. Decisions
for new features result from the requirements of the companies and individuals that use the Open Source
Job Scheduler.
If you have implemented the Job Scheduler for your company, or if you intend to do so shortly, you also will
profit from the ongoing open source development. However, it is strictly your
feedback that makes possible the rapid development of the Job Scheduler.
Therefore we urge you to take your decision for the Open Source concept by allocating budgets for the development
of new features or for the purchase of
licences. Also we require your help to develop the roadmap, it is your feedback that contributes to the
prioritisation of the features to be developed next.
Tell us your opinion about the features in planning, what priority you would give to the development and tell us about the features
you would like.
We are interested in your comments, please send your response to
info@sos-berlin.com
|
|
