Interacting with a simple storage service
The following operations are supported by an instance of
wrench::SimpleStorageService
:
Synchronously check that a file exists
Synchronously read a file (rarely used by an execution controller but included for completeness)
Synchronously write a file (rarely used by an execution controller but included for completeness)
Synchronously delete a file
Synchronously copy a file from one storage service to another
Asynchronously copy a file from one storage service to another
The first 4 interactions above are done by calling member functions of
the wrench::StorageService
class. The last two are done via a Data
Movement Manager, i.e., by calling member functions of the
wrench::DataMovementManager
class. Some of these member functions
take an optional wrench::FileRegistryService
argument, in which case
they will also update entries in a file registry service (e.g., removing
an entry when a file is deleted).
Several interactions with a simple storage service are done simple by calling
static methods of the wrench::StorageService
class. These make
it possible to lookup, delete, read, and write files. For instance:
std::shared_ptr<wrench::SimpleStorageService> storage_service;
// Get the file registry service
std::shared_ptr<wrench::FileRegistryService> file_registry;
std::shared_ptr<wrench::DataFile> some_file;
[...]
// Check whether the storage service holds the file at path /data/ and delete it if so
auto file_location = wrench::FileLocation::LOCATION(storage_service, "/data/", some_file);
if (wrench::StorageService::lookupFile(file_location))
{
std::cerr << "File found!" << std::endl;
wrench::StorageService::deleteFile(file_location, file_registry);
}
Note that the file registry service is passed to the
wrench::StorageService::deleteFile()
method since the file deletion
should cause the file registry to remove one of its entries.
Reading and writing files is something an execution controller typically does not do directly (instead, jobs created by the controller contain actions/tasks that read and write files as they execute). But, if for some reason an execution controller needs to spend time doing file I/O, it is easily done:
// Read some file from the "/" path at some storage service.
// This does not change the simulation state besides simulating a time overhead during which the execution controller is busy
wrench::StorageService::readFile(wrench::FileLocation::LOCATION(storage_service, "/", some_file));
// Write some file to the "/stuff/" path at some storage service.
// This simulates a time overhead after which the storage service will host the file. It
// is a good idea to then add an entry to the file registry service
wrench::StorageService::writeFile(wrench::FileLocation::LOCATION(storage_service, "/stuff/", some_file));
An operation commonly performed by an execution controller is copying
files between storage services (e.g., to enforce some data locality).
This is typically done by specifying file copy operations as part of
standard jobs.
But it can also be done manually by the execution controller via the
data movement manager’s methods
wrench::DataMovementManager::doSynchronousFileCopy()
and
wrench::DataMovementManager::initiateAsynchronousFileCopy()
. Here is
an example in which a file is copied between storage services:
// Create a data movement manager
auto data_movement_manager = this->createDataMovementManager();
// Synchronously copy some_file from storage_service1 to storage_service2
// While this is taking place, the execution controller is busy
data_movement_manager->doSynchronousFileCopy(wrench::FileLocation::LOCATION(storage_service1, some_file), wrench::FileLocation::LOCATION(storage_service2, some_file));
// Asynchronously copy some_file from storage_service2 to storage_service3
data_movement_manager->initiateAsynchronousFileCopy(wrench::FileLocation::LOCATION(storage_service2, some_file), wrench::FileLocation::LOCATION(storage_service3, some_file));
// Wait for and process the next event (may be a file copy completion or failure)
this->waitForAndProcessNextEvent();
See the execution controller implementation in
examples/workflow_api/basic-examples/bare-metal-data-movement/DataMovementWMS.cpp
for a more complete example.