A helper daemon (co-located with and explicitly started by a WMS), which is used to handle all job executions. More...

#include <JobManager.h>

Inheritance diagram for wrench::JobManager:
wrench::Service wrench::S4U_Daemon

Public Member Functions

 ~JobManager () override
 Destructor, which kills the daemon (and clears all the jobs)
 
std::shared_ptr< PilotJobcreatePilotJob ()
 Create a pilot job. More...
 
std::shared_ptr< StandardJobcreateStandardJob (std::vector< WorkflowTask * > tasks, std::map< WorkflowFile *, std::shared_ptr< FileLocation > > file_locations)
 Create a standard job. More...
 
std::shared_ptr< StandardJobcreateStandardJob (std::vector< WorkflowTask * > tasks, std::map< WorkflowFile *, std::shared_ptr< FileLocation > > file_locations, std::vector< std::tuple< WorkflowFile *, std::shared_ptr< FileLocation >, std::shared_ptr< FileLocation > >> pre_file_copies, std::vector< std::tuple< WorkflowFile *, std::shared_ptr< FileLocation >, std::shared_ptr< FileLocation > >> post_file_copies, std::vector< std::tuple< WorkflowFile *, std::shared_ptr< FileLocation > >> cleanup_file_deletions)
 Create a standard job. More...
 
std::shared_ptr< StandardJobcreateStandardJob (WorkflowTask *task, std::map< WorkflowFile *, std::shared_ptr< FileLocation > > file_locations)
 Create a standard job. More...
 
std::set< std::shared_ptr< PilotJob > > getPendingPilotJobs ()
 Get the list of currently pending pilot jobs. More...
 
std::set< std::shared_ptr< PilotJob > > getRunningPilotJobs ()
 Get the list of currently running pilot jobs. More...
 
void kill ()
 Kill the job manager (brutally terminate the daemon, clears all jobs)
 
void stop () override
 Stop the job manager. More...
 
void submitJob (std::shared_ptr< WorkflowJob > job, std::shared_ptr< ComputeService > compute_service, std::map< std::string, std::string > service_specific_args={})
 Submit a job to compute service. More...
 
void terminateJob (std::shared_ptr< WorkflowJob > job)
 Terminate a job (standard or pilot) that hasn't completed/expired/failed yet. More...
 
- Public Member Functions inherited from wrench::Service
void assertServiceIsUp ()
 Throws an exception if the service is not up. More...
 
std::string getHostname ()
 Get the name of the host on which the service is / will be running. More...
 
double getMessagePayloadValue (std::string)
 Get a message payload of the Service as a double. More...
 
double getNetworkTimeoutValue ()
 Returns the service's network timeout value. More...
 
bool getPropertyValueAsBoolean (std::string)
 Get a property of the Service as a boolean. More...
 
double getPropertyValueAsDouble (std::string)
 Get a property of the Service as a double. More...
 
std::string getPropertyValueAsString (std::string)
 Get a property of the Service as a string. More...
 
unsigned long getPropertyValueAsUnsignedLong (std::string)
 Get a property of the Service as an unsigned long. More...
 
bool isUp ()
 Returns true if the service is UP, false otherwise. More...
 
void resume ()
 Resume the service. More...
 
void setNetworkTimeoutValue (double value)
 Sets the service's network timeout value. More...
 
void setStateToDown ()
 Set the state of the service to DOWN.
 
void start (std::shared_ptr< Service > this_service, bool daemonize, bool auto_restart)
 Start the service. More...
 
void suspend ()
 Suspend the service.
 
- Public Member Functions inherited from wrench::S4U_Daemon
 S4U_Daemon (std::string hostname, std::string process_name_prefix, std::string mailbox_prefix)
 Constructor (daemon with a mailbox) More...
 
virtual ~S4U_Daemon ()
 
virtual void cleanup (bool has_returned_from_main, int return_value)
 Cleanup function called when the daemon terminates (for whatever reason). The default behavior is to throw an exception if the host is off. This method should be overriden in a daemons implements some fault-tolerant behavior, or is naturally tolerant. More...
 
void createLifeSaver (std::shared_ptr< S4U_Daemon > reference)
 Create a life saver for the daemon. More...
 
std::string getName ()
 Retrieve the process name. More...
 
int getReturnValue ()
 Returns the value returned by main() (if the daemon has returned from main) More...
 
S4U_Daemon::State getState ()
 Get the daemon's state. More...
 
bool hasReturnedFromMain ()
 Returns true if the daemon has returned from main() (i.e., not brutally killed) More...
 
bool isDaemonized ()
 Return the daemonized status of the daemon. More...
 
bool isSetToAutoRestart ()
 Return the auto-restart status of the daemon. More...
 
std::pair< bool, int > join ()
 Join (i.e., wait for) the daemon. More...
 
void resumeActor ()
 Resume the daemon/actor.
 
void setupOnExitFunction ()
 Sets up the on_exit functionf for the actor.
 
void startDaemon (bool daemonized, bool auto_restart)
 Start the daemon. More...
 
void suspendActor ()
 Suspend the daemon/actor.
 

Protected Member Functions

 JobManager (std::shared_ptr< WMS > wms)
 Constructor. More...
 
- Protected Member Functions inherited from wrench::Service
 Service (std::string hostname, std::string process_name_prefix, std::string mailbox_name_prefix)
 Constructor. More...
 
 ~Service ()
 Destructor.
 
template<class T >
std::shared_ptr< T > getSharedPtr ()
 Method to retrieve the shared_ptr to a service. More...
 
void serviceSanityCheck ()
 Check whether the service is properly configured and running. More...
 
void setMessagePayload (std::string, double)
 Set a message payload of the Service. More...
 
void setMessagePayloads (std::map< std::string, double > default_messagepayload_values, std::map< std::string, double > overriden_messagepayload_values)
 Set default and user-defined message payloads. More...
 
void setProperties (std::map< std::string, std::string > default_property_values, std::map< std::string, std::string > overriden_property_values)
 Set default and user-defined properties. More...
 
void setProperty (std::string, std::string)
 Set a property of the Service. More...
 
- Protected Member Functions inherited from wrench::S4U_Daemon
void acquireDaemonLock ()
 Lock the daemon's lock.
 
void killActor ()
 Kill the daemon/actor (does nothing if already dead) More...
 
void releaseDaemonLock ()
 Unlock the daemon's lock.
 
void runMainMethod ()
 Method that run's the user-defined main method (that's called by the S4U actor class)
 

Additional Inherited Members

- Public Types inherited from wrench::S4U_Daemon
enum  State { UP, DOWN, SUSPENDED }
 Daemon states. More...
 
- Static Public Member Functions inherited from wrench::Service
static void cleanupTrackedServices ()
 Go through the tracked services and remove all entries with a refcount of 1!
 
static void clearTrackedServices ()
 Forget all tracked services.
 
static void increaseNumCompletedServicesCount ()
 Increase the completed service count.
 
- Public Attributes inherited from wrench::S4U_Daemon
std::string hostname
 The name of the host on which the daemon is running.
 
std::string initial_mailbox_name
 The initial name of the daemon's mailbox.
 
LifeSaver * life_saver = nullptr
 The daemon's life saver.
 
std::string mailbox_name
 The current name of the daemon's mailbox.
 
std::string process_name
 The name of the daemon.
 
Simulationsimulation
 a pointer to the simulation object
 
- Static Protected Member Functions inherited from wrench::Service
static void assertServiceIsUp (std::shared_ptr< Service > s)
 Assert for the service being up. More...
 
template<class T >
static std::shared_ptr< T > getServiceByName (std::string name)
 Method to retrieve the shared_ptr to a service based on the service's name (not efficient) More...
 
- Protected Attributes inherited from wrench::Service
std::map< std::string, double > messagepayload_list
 The service's messagepayload list.
 
std::string name
 The service's name.
 
double network_timeout = 30.0
 The time (in seconds) after which a service that doesn't send back a reply (control) message causes a NetworkTimeOut exception. (default: 30 second; if <0 never timeout)
 
std::map< std::string, std::string > property_list
 The service's property list.
 
- Protected Attributes inherited from wrench::S4U_Daemon
unsigned int num_starts = 0
 The number of time that this daemon has started (i.e., 1 + number of restarts)
 
State state
 The service's state.
 

Detailed Description

A helper daemon (co-located with and explicitly started by a WMS), which is used to handle all job executions.

Constructor & Destructor Documentation

◆ JobManager()

wrench::JobManager::JobManager ( std::shared_ptr< WMS wms)
explicitprotected

Constructor.

Parameters
wmsthe wms for which this manager is working

Member Function Documentation

◆ createPilotJob()

std::shared_ptr< PilotJob > wrench::JobManager::createPilotJob ( )

Create a pilot job.

Returns
the pilot job
Exceptions
std::invalid_argument

◆ createStandardJob() [1/3]

std::shared_ptr< StandardJob > wrench::JobManager::createStandardJob ( std::vector< WorkflowTask * >  tasks,
std::map< WorkflowFile *, std::shared_ptr< FileLocation > >  file_locations 
)

Create a standard job.

Parameters
tasksa list of tasks (which must be either READY, or children of COMPLETED tasks or of tasks also included in the list)
file_locationsa map that specifies locations where files, if any, should be read/written. When empty, it is assumed that the ComputeService's scratch storage space will be used.
Returns
the standard job
Exceptions
std::invalid_argument

◆ createStandardJob() [2/3]

std::shared_ptr< StandardJob > wrench::JobManager::createStandardJob ( std::vector< WorkflowTask * >  tasks,
std::map< WorkflowFile *, std::shared_ptr< FileLocation > >  file_locations,
std::vector< std::tuple< WorkflowFile *, std::shared_ptr< FileLocation >, std::shared_ptr< FileLocation > >>  pre_file_copies,
std::vector< std::tuple< WorkflowFile *, std::shared_ptr< FileLocation >, std::shared_ptr< FileLocation > >>  post_file_copies,
std::vector< std::tuple< WorkflowFile *, std::shared_ptr< FileLocation > >>  cleanup_file_deletions 
)

Create a standard job.

Parameters
tasksa list of tasks (which must be either READY, or children of COMPLETED tasks or of tasks also included in the standard job)
file_locationsa map that specifies locations where input/output files, if any, should be read/written. When empty, it is assumed that the ComputeService's scratch storage space will be used.
pre_file_copiesa vector of tuples that specify which file copy operations should be completed before task executions begin. The ComputeService::SCRATCH constant can be used to mean "the scratch storage space of the ComputeService".
post_file_copiesa vector of tuples that specify which file copy operations should be completed after task executions end. The ComputeService::SCRATCH constant can be used to mean "the scratch storage space of the ComputeService".
cleanup_file_deletionsa vector of file tuples that specify file deletion operations that should be completed at the end of the job. The ComputeService::SCRATCH constant can be used to mean "the scratch storage space of the ComputeService".
Returns
the standard job
Exceptions
std::invalid_argument

◆ createStandardJob() [3/3]

std::shared_ptr< StandardJob > wrench::JobManager::createStandardJob ( WorkflowTask task,
std::map< WorkflowFile *, std::shared_ptr< FileLocation > >  file_locations 
)

Create a standard job.

Parameters
taska task (which must be ready)
file_locationsa map that specifies locations where input/output files should be read/written. When unspecified, it is assumed that the ComputeService's scratch storage space will be used.
Returns
the standard job
Exceptions
std::invalid_argument

◆ getPendingPilotJobs()

std::set< std::shared_ptr< PilotJob > > wrench::JobManager::getPendingPilotJobs ( )

Get the list of currently pending pilot jobs.

Returns
a set of pilot jobs

◆ getRunningPilotJobs()

std::set< std::shared_ptr< PilotJob > > wrench::JobManager::getRunningPilotJobs ( )

Get the list of currently running pilot jobs.

Returns
a set of pilot jobs

◆ stop()

void wrench::JobManager::stop ( )
overridevirtual

Stop the job manager.

Exceptions
WorkflowExecutionException
std::runtime_error

Reimplemented from wrench::Service.

◆ submitJob()

void wrench::JobManager::submitJob ( std::shared_ptr< WorkflowJob job,
std::shared_ptr< ComputeService compute_service,
std::map< std::string, std::string >  service_specific_args = {} 
)

Submit a job to compute service.

Parameters
joba workflow job
compute_servicea compute service
service_specific_argsarguments specific for compute services:
  • to a BareMetalComputeService: {{"taskID", "[hostname:][num_cores]}, ...}
    • If no value is provided for a task, then the service will choose a host and use as many cores as possible on that host.
    • If a "" value is provided for a task, then the service will choose a host and use as many cores as possible on that host.
    • If a "hostname" value is provided for a task, then the service will run the task on that host, using as many of its cores as possible
    • If a "num_cores" value is provided for a task, then the service will run that task with this many cores, but will choose the host on which to run it.
    • If a "hostname:num_cores" value is provided for a task, then the service will run that task with the specified number of cores on that host.
  • to a BatchComputeService: {{"-t":"<int>" (requested number of minutes)},{"-N":"<int>" (number of requested hosts)},{"-c":"<int>" (number of requested cores per host)}[,{"-u":"<string>" (username)}]}
  • to a VirtualizedClusterComputeService: {} (jobs should not be submitted directly to the service)}
  • to a CloudComputeService: {} (jobs should not be submitted directly to the service)}
  • to a HTCondorComputeService:
    • For a "grid universe" job that will be submitted to a child BatchComputeService: {{"universe":"grid", {"-t":"<int>" (requested number of minutes)},{"-N":"<int>" (number of requested hosts)},{"-c":"<int>" (number of requested cores per host)}[,{"-service":"<string>" (batch service name)}, {"-u":"<string>" (username)}]}
    • For a "non-grid universe" job that will be submitted to a child BareMetalComputeService: {}
Exceptions
std::invalid_argument
WorkflowExecutionException

◆ terminateJob()

void wrench::JobManager::terminateJob ( std::shared_ptr< WorkflowJob job)

Terminate a job (standard or pilot) that hasn't completed/expired/failed yet.

Parameters
jobthe job to be terminated
Exceptions
WorkflowExecutionException
std::invalid_argument
std::runtime_error

The documentation for this class was generated from the following files: