OPEN SOURCE - Job Scheduler - Solutions

• Creating a Job
• What is the concept of "job chains and order processing"?
• Could you show me a simple example of a job chain?
• Job Initiation
  
  

• Licence Key for the Job Scheduler
• eMails cannot be sent
• Database Connections to MySQL get lost
• Why does the log file show on GUI with delay?
• Why does the Job Scheduler slow down the execution of jobs?
  
  

• Make the Job Scheduler work with MySQL 4.0.x
• JDBC Connection to SQL Server
• Database Connections to MySQL get lost
• How can we setup our log files to be written in a database?
  
  

• When to use the open source and the closed source licence
• Handling of support options for multiple Job Scheduler installations
  
  

• Job Dependencies
• Scheduling for Multiple Jobs
• Parameter Sets for Multiple Jobs
• Job Streams
• Notification on Job Completion
• Delayed and Repeated Processing
• Viewing Job Schedules
• Ad-Hoc Changes to Job Streams
• Trigger Error Conditions
• Drag & Drop for Job Configuration
• Viewing Jobs
• User-friendly drop-down Environment
• Built in Utilities for Routing Job Output
• Start Job by Notification
• Run Jobs as a Specific Userid
• Control and Monitor Job Executions via Web Interface
• Import Job Specifications from Text Implementation
• Run in Load Balanced or Fault Tolerance Mode
• Run Job Agents on Multiple Machines
• Script Jobs in Standard Unix Scripting Languages
• Restart of the Job Scheduler
• Organizing secure file transfer with the Job Scheduler
• How to implement Jobs with Perl?
• Restoring enqueued orders and jobs in job chains
• How can we configure a holiday setting for the Job Scheduler?
• The Master and Slave relationship of the Job Scheduler
• When do changes to a Job Chain apply?
• How to execute Jobs and Orders in remote Job Scheduler Instances?
• What is the difference between directory monitoring and file orders?
• How to embed double quotes in a job or order parameter?
• How to run a task manually standalone, that is shown as part of a job chain?
• How can Job Scheduler run a program that shows a GUI?
• Can Job Scheduler be operated for 32bit and 64bit environments?

<?xml version="1.0" encoding="iso-8859-1"?>
<spooler>
  <config>
    <jobs>

       <job name = "my_shell_script">
          <process file  = "my_shell_script.sh"
                   param = ""/>
          <run_time repeat = "3600"
                    begin  = "00:00"
                    end    = "24:00"/>
       </job>

     </jobs>
   </config>
</spooler>
              

1. Using job chains
   
   Sequences of jobs can easily be configured in job chains that guarantee that a successor job is not invoked if a previous job provokes an error. Job errors can be configured to be detected by execution code (for executable files), signals (for Unix only), availability of output to stderr and by errors that were raised either by API methods or by SQL exceptions (and normal SQL errors).
   
   Job chains are nodes of jobs that are organized in a sequence, every node is given a distinct state. An order proceeds along the jobs in a job chain one after the other. Should a processing error occur, then the order is stopped and removed from the job chain.
   See the samples in What is the concept of "job chains and order processing"?
   
   In addition to this you could use monitor scripts to implement more complex conditions for job starts in any of the languages Java, Javascript, Perl and VBscript. The Job Scheduler API provides a method scheduler_job.start() that can be used in scripts that implement their own business logic for conditional job starts.
   
   Moreover this API method can be exposed to Oracle pl/sql procedures or functions (by Java classes in the database) that enable jobs to be launched in a transactional environment (e.g. start a job exlusively after commit has been invoked) either by triggers or by pl/sql procedures.
   
   
2. Using successor jobs
   
   You could configure any jobs to succeed the current job based on it's execution result:

   <job>
     <process file="..."> or <script language="...">
       <!-- in case of success the exit code=0, let's start this job -->
       <commands on_exit_code="success">
         <start_job job="ftp_get_files"><params/></start_job>
       </commands>
       <!-- in case of error the exit code!=0 -->
       <commands on_exit_code="error">...</commands>
       <!-- other exit codes could be handled individually -->
       <commands on_exit_code="1 2 4 8">...</commands>
     </process>
   </job>
   

   You could configure any job to start arbitrary successor jobs and orders from Job Scheduler release 1.2.7 on.
   
   

• file system events: file has been added deleted; file names to be triggered can be configured as regular expressions;
• calendar events: a given date, weekday, day of month, ultimo; these events can be configured to respect holidays;
• API events: use of the API method spooler_job.start(); this method can be used by other jobs in any of the languages Java, Javascript, Perl, VBScript;
• XML commands sent to the Job Scheduler via TCP/IP or UDP datagrams. Jobs can be started, stopped (and otherwise controlled) by simple XML commands, e.g.
  <start_job job="cleanup_tmp_files" at="now">.
• events created by the web interface of the Job Scheduler: these events use the above XML commands.

<job name = "cleanup_sos_files" title = "Remove temporary files">
  <params>
    <!-- remove temporary files from the given directory otherwise 
    from the users temp directory  --> 
    <param name = "file_path" value = "/tmp"/> 
    <!-- remove temporary files with the given prefix --> 
    <param name = "file_specification" value = "^(sos.*)"/> 
  </params> 
  <script language = "java"
          java_class = "sos.scheduler.job.JobSchedulerCleanupFiles"/> 

  <!-- cleanup files in the given interval of seconds (4 hours) --> 
  <run_time let_run = "yes"
            begin   = "06:00"
            end     = "22:00"
            repeat  = "04:00:00"/> 
</job> 


<job name = "cleanup_tmp_files" title = "Remove temporary files"/> 
  <params/> 
    <param name = "file_path" value = "/tmp"/> 
    <param name = "file_specification" value = "^(tmp.*)"/> 
  </params/> 

  <script language = "java"
          java_class = "sos.scheduler.job.JobSchedulerCleanupFiles"/> 

  <!-- cleanup files in the given interval of seconds (24 hours)--> 
  <run_time let_run= "yes"
            begin   = "00:00"
            end     = "24:00"
            repeat  = "24:00:00"/> 
</job> 
              

• mail_on_error
• mail_on_warning
• mail_on_success
• mail_on_process (reserved for API based jobs)

• A simple way to make a job sleep and wake up is to configure a repeat interval in hours, minutes and seconds. In this case the job is not exactly "delayed" but will be restarted as soon as the interval has elapsed.
  
  
• A job can delay its processing by the Job Scheduler API method delay_spooler_process(seconds_or_hhmmss). This is available only if the job implements the method spooler_process() of the API. In this case the job remains loaded, i.e. it is in processing but uses no CPU until the scheduled interval has elapsed.
  
  
• A job can be configured for restart should an error have occurred, for example by configuration or API methods:
  
  spooler_job.delay_after_error( 2 ) = 10; // 10 seconds
spooler_job.delay_after_error( 5 ) = "00:01"; // one minute
spooler_job.delay_after_error( 10 ) = "24:00"; // one day
spooler_job.delay_after_error( 20 ) = "STOP";
  
  In this example, the Job Scheduler repeats a job immediately after a (first) error occurs. After two to four consecutive errors, however, the Job Scheduler delays restarting the job by 10 seconds; between five and nine consecutive errors, the delay is one minute; between ten and nineteen errors, it is 24 hours. The job is stopped after the twentieth consecutive error.
  
  It is possible to set the value of the seconds_or_hhmm_ss parameter to "STOP" in order to restrict the number of (unsuccessful) repetitions of a job. The job then is stopped, i.e. will not restart automatically, when the number of consecutive errors specified is reached.
  
  

<show_state job="scheduler_cleanup_sos_files" what="all"/>

<job job="scheduler_rotate_log" state="pending" title="Rotate 
  and compress logfiles"
     all_steps="0" all_tasks="0"
     log_file="c:/scheduler/logs/job.scheduler_rotate_log.log"
     order="no" tasks="1" in_period="yes" next_start_time="2005-10-18 01:00:00">
  <tasks count="0"/> 
  <queued_tasks length="0"/> 
  <log level="info" highest_level="debug3" last_info=""
       mail_on_error="yes"
       mail_on_warning="yes" smtp="localhost"
       mail_from="scheduler@sos-berlin.com"
       mail_to="info@sos-berlin.com"/> 
</job>

    <job stop_on_error="no">
    ...
      <commands on_exit_code="error">
        <start_job job="my_error_handler"><params/></start_job>
      </commands>
    </job>
               

<security ignore_unknown_hosts = "yes">
  <allowed_host host = "192.11.0.0"  level = "info"/>
  <allowed_host host = "192.11.0.99" level = "none"/>
  <allowed_host host = "192.11.0.27" level = "all"/>
</security>

• All hosts in the network 192.11.0.0 have read access to the scheduler interface, i.e. logs and schedules
• The host 192.11.0.99 is not allowed any access
• The host 192.11.0.27 is allowed full access, i.e. start jobs and orders, remove jobs etc.

• to stdout
• to stderr
• created by API logging methods (info, warning, error, debug)
• created by SQL errors and by usage of the Oracle Package dbms_output
• to log files external to the Job Scheduler but for which a path was configured with the job

<job name = "sample_job">
  <process file  = "/home/test/scheduler/jobs/sample_script.pl"
           param = "-f ${SCHEDULER_TASK_TRIGGER_FILES}"/>
  <start_when_directory_changed directory = "/tmp/input" regex = ".*"/>
</job>

• Use the executable setuid
  
  we provide the executable setuid for Unix (source code available) that is used to launch executable files on the local machine in a different account. No passwords for target accounts have to be provided. The program setuid should be configured with the following permissions:
  
  cd ~/scheduler/bin
  mkdir nobody
  chmod go-rwx nobody
  cp setuid nobody/setuid-nobody
  chown nobody nobody/setuid-nobody (has to be done by root)
  chmod u+s,a-r nobody/setuid-nobody (has to be done by root)
  
  
  Suppose the Job Scheduler is operated in the account "test" and has to launch executables in the account "nobody": with the above commands you will create a write-protected directory and copy of setuid for the account "nobody". The file setuid-nobody is executable for the account "test" but is owned by "nobody". With setuid-nobody the Job Scheduler can run any commands for the account "nobody" without giving a password for the target account. Therefore setuid-nobdy checks if the above permissions are set and otherwise terminates.
  
  For this solution you should ensure that the Job Scheduler's account and password are not publicly available to other users who are not authorized to run executables in the target account.
  
  A sample configuration in the file scheduler.xml for a command that is executed in the target account might look like this:

  
  <job name = "sample_setuid_job">
    <process file  = "$SCHEDULER_HOME/bin/nobody/setuid-nobody"
             param = "/home/test/scheduler/jobs/sample_script_notification.pl"/>
  </job>
  

• Use secure shell
  
  Standalone scripts can be processed by ssh. You would have to:
  • set up the shh client and server
  • create a private/public key pair for the Job Scheduler on the client machine and publish your public key to the ssh server
  • run jobs via <process file="ssh"/> and give the ssh command line parameters in the attribute "param" of the <process> element
  
  ssh can be used to launch executables on your local and on remote machines. Alternatively you could install a Job Scheduler instance (=agent) on the remote machine which is less overhead if there is no ssh and pki infrastructure.
  
  

• Job Errors: jobs may be configured to repeat automatically after errors in configurable intervals and number of trials.
  
  
• Job Scheduler Errors: if the scheduler is operated with a database (which is not required) then automatic reconnects are available with a configurable number of trials. there are mechanisms to
  • Monitor errors, i.e. multiple Job Scheduler instances on the same or different servers send heartbeats to a Job Scheduler that is operated exclusively for monitoring;
  • Respawn a faulty Job Scheduler instance;
  • Do some "sanity checking" for disk space and memory usage by maintenance jobs.
    
    
  Fault tolerance is not available in the sense of automatic failover: if resources (disk space,
  database etc.) run short, then the Job Scheduler will finally shutdown. There is no such thing yet as a backup Job Scheduler instance starting automatically on a different server to continue jobs that were abandoned by the faulting Job Scheduler instance.
  
  

• to write to the Job Scheduler logs which are visible in the web interface (and automatically mailed in case of errors, warnings or success),
• to set the job processing state,
• to launch other jobs via API and to process job parameters.

• Java,
• Javascript, the Mozilla (see SpiderMonkey) runtime environment is included for the above platforms,
• for Windows additionally VBScript is supported.
• Perl jobs with API support may be operated for Windows provided that an ActiveState implementation is installed. For Linux and Solaris there are restrictions due to the fact that Perl is not threadsafe implemented, for both platforms we have to link the Perl libraries to the scheduler.
  
  

1. The remote server operating system is not supported by the Job Scheduler.
   
   
2. We don't wish to rely on a program that uses the Job Scheduler API package running on the remote node to send us a notification (we can't maintain programs running on this remote node).
   
   

1. Proposed Solution: Signalling
   
   Should the remote host be responsible to signal the existence of a file ready for transfer, then concerning the remote host
   • don't be afraid of the Job Scheduler API, it might be as simple as this:
     
     telnet [scheduler_host] [scheduler_port]
<start_job job="ftp_get_files"></params></start_job>
     
     This command is most likely available for the remote host and its simple: by means of a TCP stack you could send an XML command that tells the Job Scheduler what should be done, e.g. to start a job. Major parts of the API are available as XML commands that could be sent via TCP or UDP to the Job Scheduler. You do not have to use telnet as the ultimate TCP solution, it's just a sample that works at the command line.
     
     
   • use the Job Scheduler API without installation
     we deliver Java classes that could be used without a Job Scheduler being installed, e.g. on the command line like this:
     
     java -cp ./sos.scheduler.jar:./sos.util.jar sos.scheduler.SOSSchedulerCommand [scheduler_host] [scheduler_port] [xml command]
     
     Java is available for most operating systems, therefore the remote host could use this command line to send any XML command to the Job Scheduler without the need to have a Job Scheduler being installed on the remote host. A standard JRE 1.4.2++ and the above .jar archives would be sufficient for the remote host.
     
     
   • use an alternative TCP/UDP stack
     You could use any programming/scripting language that supports TCP or UDP to send XML commands.
     
     
   The above cases do not assume that you have to maintain the resp. scripts/programs on the remote host as standard operating system utilities could be used for this. We recommend to use UDP in the above solutions as this protocol is non-blocking, i.e. the remote host will not suffer (and have to recover) from errors due to network downtimes. Therefore use of signalling should be optional in the communication between remote host and Job Scheduler to speed up processing but is no replacement for polling.
   
   
2. Proposed solution: polling
   
   we recommend this solution as it is save and easy to implement to check if a file exists on a remote host:
   
   • Check via FTP
     See the standard jobs in the chapter for FTP processing at Job Scheduler Standard Jobs and the job documentation in Job Scheduler FTP Receive. The existing code could be adapted to return distinct exit codes that signal if files are available or absent in a remote directory.
     
     
   • Check via ssh
     This solution requires a shell script or operating system command on the remote host that is launched via ssh by a job in order to check if a file is ready to be transferred. The script/command could be implemented to return a distinct exit code which signals that files are ready for transfer.
     
     A pseudo configuration for scheduler.xml could be:

     <job>
       <process file="ssh" param="-l [user]@[host] [script_or_command]">
         <!-- restart this job if an error occurs, e.g. file not present, 
              server unavailable etc. -->
         <delay_after_error error_count="1" delay="60"/> 
         <!-- repeat this job every 60s after the first error occurred -->
         <delay_after_error error_count="10" delay="600"/> 
         <!-- repeat this job every 600s after the 10th error occurred -->
         <delay_after_error error_count="50" delay="stop"/> 
         <!-- stop this job after 50 subsequent errors -->
         <!-- in case of files found: exit code=0, let's fetch the files -->
         <commands on_exit_code="success">
         <start_job job="ftp_get_files"><params/></start_job></commands>
         <!-- in case of absence of files: exit code=1 -->
         <!-- let's ignore this return code -->
         <commands on_exit_code="1"/>
         <!-- other exit codes are automatically handled as errors, 
              otherwise you could start individual jobs to handle errors -->
         <!-- <commands on_exit_code="error">...</commands> -->
       </process>
     </job>
     

     Starting successor jobs based on exit codes is not restricted to ssh or the use of <process file=""/>, you could configure any job to start arbitrary successor jobs and orders from Job Scheduler release 1.2.7 on.
     
     
3. Concurrency Issues
   
   A simple way to avoid concurrency would be to have the client put files with a name as filename~ and rename them to filename. This would make the files appear atomically.
   
   More sophisticated solutions are needed in the following cases:
   • What should happen if one input file cannot be fetched for a longer period and subsequent files are provided by the remote host?
   • How should multiple input files be handled that are to be processed in a single transaction by the client, e.g. for dbms import?
     
     
   We implemented commercial solutions for these issues based on traffic light rules. If this should be of any interest to you then please contact us.
   
   

• for Windows we do not distribute a library but use the scripting libraries from an existing ActiveState Perl implementation, see http://www.activestate.com.
• for Unix the library libsosperlscript.so should have been copied to the lib directory by the installer. Moreover the library libperl.so is given in this directory: this library could be replaced from your Perl installation, it is the subprogram interface that must fit to the threading options that Perl was compiled with. The version that is provided by the setup is for a multithreaded Perl 5.8.0.
  
  

• you could use
  <job><process file="my_script.pl"/></job>
  to use Perl as from the command line (my_script.pl must be executable). This works with all versions of Perl, but no API methods of the Job Scheduler can be used.
• you would use
  <job><script language="perlscript"><include file="my_api_script.pl"/></job>
  for a Perl script that makes use of the Job Scheduler API and requires the above libraries for Unix.
  
  

• via the built-in web interface of the Job Scheduler
• at http://www.sos-berlin.com/doc/en/scheduler.doc/api/api-perl.xml

  # to make the object variables available ...
  use vars qw ($spooler $spooler_log $spooler_job $spooler_task);
  
  # ... then use the objects and methods, e.g to access a job parameter
  $spooler_task->params->value("request_stylesheet")

  <spooler>
    <config>
      <jobs>
      ...       
      </jobs>
      <job_chains>
        <job_chain name="TestGetandPut">
        <job_chain_node state="1"     
            job="TestGet"     
            next_state="100"     
            error_state="999"/> 
        <job_chain_node state="100" 
            job="GetPut"     
            next_state="200"     
            error_state="999"/> 
        <job_chain_node state="200" /> 
        <job_chain_node state="999" />
      </job_chain>
       ...      
      </job_chains>
      <commands>
        <!-- here you add an order for your job chain>
        <add_order job_chain=" TestGetandPut" 
        title="sample order" replace="yes">
          <params/>
            <!-- sample run time: repeat the execution of this order 
                every day at the given hour -->
            <!-- without run time specification an order is fired just 
                once and will not be repeated -->    
            <run_time><period single_start="23:36"/></run_time>
        </add_order>
      </commands>
    </config>
  </spooler>

• Orders can be processed simultaneously, the jobs can be configured (e.g. by <job tasks="3"/>) to run in parallel tasks to process these orders.
• Orders can have parameters, i.e you use the same job implementation that is parameterized at execution time by the respective order, e.g. the order could contain the settings required for ftp connections.
• Orders could have a payload, i.e. arbitrary xml element structures that are accessible by every job in a job chain and can thus be used to pass execution results from one job in a job chain to the next one.
• Orders can be set back, i.e. could change their position in the job chain to be repeated starting with a distinct job chain node.
• Every order has its own log file that contains the information of all jobs that have been passed in the job chain, whereas subsequent tasks would create individual task logs.

• Let's look at the job implementation. Any job could be used for a job chain, however the attribute order="yes" is required:
  
  

    <spooler>
      <config>
        <jobs>
          <job name="command_1" title="sample_command" order="yes">
            <script language="shell">
              <![CDATA[
               C:\Programme\Samples\Scripts\command_sample_1.cmd
              ]]>
            <script>
          </job>
  
          <job name="command_2" title="sample_command" order="yes">
            <script language="shell">
              <![CDATA[
               C:\Programme\Samples\Scripts\command_sample_2.cmd
              ]]>
            <script>
          </job>
        </jobs>
      </config>
    </spooler>
  

• Next you need a job chain that assigns nodes and states to these jobs:
  
  

    <spooler>
      <config>
        <job_chains>
          <job_chain name="command_sequence">
            <job_chain_node state="first"     
                job="command_1"     
                next_state="second"     
                error_state="error"/> 
            <job_chain_node state="second" 
                job="command_2"     
                next_state="success"     
                error_state="error"/> 
            <job_chain_node state="success"/> 
            <job_chain_node state="errror"/>
          </job_chain>
        </job_chains>
      </config>
    </spooler>
  

  This sample assigns the above job command_1 the first node in the chain and command_2 the second node that will be executed if the first node were successfully completed. The chaining is organized by the next_state attribute that identifies the state of the next job node in a chain and respectively the error_state attribute that identifies the node that should be executed in case of error. Job nodes for final states could be empty as for success and error, moreover you could create job implementations for individual error handling.
  
  
• Last but not least you need an order that triggers the execution of the job chain:
  
  

    <spooler>
      <config>
        <commands>
          <add_order job_chain="command_sequence" 
            title="sample command sequence" replace="yes">
            <params/>
              <!-- sample run time: repeat the execution of this order 
                   every day at the given hour -->
              <run_time><period single_start="23:36"/></run_time>
          </add_order>
        </commands>
      </config>
    </spooler>
  

  This order is used for three purposes:
  • Trigger the execution of the job chain:
    You would not start a job chain by starting the first job node, but by starting the order. The built-in HTML interface of the Job Scheduler offers the respective operations for orders. Multiple orders could cause the parallel execution of jobs in a job chain. The Job Scheduler would then dynamically allocate tasks in parallel to these jobs.
  • Add parameters to the jobs within the job chain:
    Parameters from orders are added with precedence to the parameters of a job configuration (<job><params><param name="sample_name" value="sample_value"></params></job>).
  • Specify the run time for automated execution:
    Any periods for repeated execution could be set.

• The method spooler_process() is left for jobs that make use of the API via <script/>
  
• An executable file has been terminated that was started via <process/>
  
  

• Tasks that are running and are killed due to killing the Job Scheduler main task are not restarted on scheduler startup (as the Job Scheduler cannot decide if it's decent to re-execute this job regarding run times and corresponding begin and end time slots etc.).
  
  
• Should a task be enqueued for a future date, e.g. with
  <start_job job="..." at="now+3600"/><params/></start_job>
  or the respective API methods then this task's start time is reflected in the database (in the table SCHEDULER_TASKS). For orders and jobs the Job Scheduler will read the respective tables on startup and enqueue the respective orders and task to the given start time.

  <config>
      <holidays>
          <holiday date="2006-12-25"/>
          ...
          <holiday date="2006-12-26"/>
      </holidays>
      <jobs/>
      <job_chains/>
  </config>

  <job>
    <run_time let_run="no">
        <period single_start = "08:00"/>
        <holidays>
            <holiday date="2006-06-19"/>
            ...
            <holiday date="2006-07-19"/>
        </holidays>
    </run_time>
  </job>              

  <config main_scheduler        = "remotehost:4444">
  ...
  </config>

  <config>
      <script language = "java"
             java_class = "sos.scheduler.managed.JobSchedulerManagedStarter"/>
      <jobs/>
      <job_chains/>
  </config>

  ;                         enable job history, if set to yes the 
  ;                         scheduler keeps a job history in 
  ;                         csv files or database tables
  history                 = yes
  ;                         store job protocol for task history 
  ;                         (yes|no|gzip, default: no)
  history_with_log        = gzip
  ;                         store job protocol for order history 
  ;                         (yes|no|gzip, default: no)
  order_history_with_log  = gzip
  ;                         store protocol for scheduler history 
  ;                        (yes|no|gzip, default: no)
  history_archive         = gzip

• If you bundle the Job Scheduler with your application for shipment to third parties
• If you need an indemnity agreement that protects you and your company from third party claims
• If you wish to have a limited warranty for two years which includes bugfixing at no additional charge; you could run the Job Scheduler without any support option and safely rely on bugs being fixed during the warranty period.
• If you choose one of the offered support options and the calculated pricing is favourable.

• Buy support options for three servers: one for Linux, one for Solaris, one for Windows. Alternatively buy support options for the respective operating systems, e.g. Win2000, Win2003, WinXP
• You will receive three support licence keys that are restricted to three Job Scheduler installations
• You operate the Job Scheduler for all other servers without support option
• When opening a support request for one of these platforms/ operating systems then the respective support licence key should be submitted to identify the underlying support option (we do not require to identify server, cpu or details that you won't submit voluntarily)
• If you want to submit a support request for a server that is not covered by one of the above support licence keys then you would have to reproduce the problem for one of the supported servers

        <process_classes>
            <process_class max_processes="20"/>
            <process_class name="many_processes" max_processes="30"/>
            ...
        </process_classes>