wrench::CompoundStorageService

class CompoundStorageService : public wrench::StorageService

An abstract storage service which holds a collection of concrete storage services (eg. SimpleStorageServices). It does not provide direct access to any storage resource. It is meant to be used as a way to postpone the selection of a storage service for a file action (read, write, copy, etc) until a later time in the simulation, rather than during job definition. A typical use for the CompoundStorageService is to select a definitive SimpleStorageService for each action of a job during its scheduling in a BatchScheduler class. This should never receive messages for I/O operations, as any standard storage service (File Read/Write/Delete/Copy/Lookup requests), instead, it overides the main functions of StorageService (readFile / writeFile /…) and will craft messages intended for one or many of its underlying storage services.

Public Functions

CompoundStorageService(const std::string &hostname, std::set<std::shared_ptr<StorageService>> storage_services, StorageSelectionStrategyCallback &allocate, WRENCH_PROPERTY_COLLECTION_TYPE property_list = {}, const WRENCH_MESSAGE_PAYLOAD_COLLECTION_TYPE &messagepayload_list = {})

Constructor.

Parameters:

  • hostname – the name of the host on which this service will run

  • storage_services – subordinate storage services

  • allocate – the storage allocation strategy

  • property_list – the configurable properties

  • messagepayload_list – the configurable message payloads

CompoundStorageService(const std::string &hostname, std::set<std::shared_ptr<StorageService>> storage_services, WRENCH_PROPERTY_COLLECTION_TYPE property_list = {}, WRENCH_MESSAGE_PAYLOAD_COLLECTION_TYPE messagepayload_list = {})

Constructor for the case where no request message (for I/O operations) should ever reach the CompoundStorageService. This use case suppose that any action making use of a FileLocation referencing this CompoundStorageService will be intercepted before its execution (in a scheduler for instance) and updated with one of the StorageServices known to this CompoundStorageService.

Parameters:

  • hostname – the name of the host on which this service will run

  • storage_services – subordinate storage services

  • property_list – the configurable properties

  • messagepayload_list – the configurable message payloads

void copyFileIamDestination(const std::shared_ptr<FileLocation> &src_location, const std::shared_ptr<FileLocation> &dst_location)

Copy file from a SimpleStorageService to a CSS. Src file cannot be stripped, but copy might result in stripped file on CSS.

Parameters:

  • src_location – the source location

  • dst_location – the destination location

void copyFileIamSource(const std::shared_ptr<FileLocation> &src_location, const std::shared_ptr<FileLocation> &dst_location)

Copy file from css to a simple storage service (file might be stripped within the CSS, but should be reassembled on the SSS)

Parameters:

  • src_location – the source location

  • dst_location – the destination location

inline void createFile(const std::shared_ptr<DataFile> &file)

Create a file at the storage service (in zero simulated time)

Parameters:

file – a file

inline void createFile(const std::shared_ptr<DataFile> &file, const std::string &path)

Create a file at the storage service (in zero simulated time)

Parameters:

  • file – a file

  • path – a path

inline virtual void createFile(const std::shared_ptr<FileLocation> &location) override

Create a file at the storage service (in zero simulated time)

Parameters:

location – a location

void createFile(const std::shared_ptr<FileLocation> &location) = 0

Create a file at the storage service (in zero simulated time)

Parameters:

location – a location

inline void deleteFile(const std::shared_ptr<DataFile> &file)

Delete a file at the storage service (incurs simulated overheads)

Parameters:

file – a file

inline void deleteFile(const std::shared_ptr<DataFile> &file, const std::string &path)

Delete a file at the storage service (incurs simulated overheads)

Parameters:

  • file – a file

  • path – a path

inline void deleteFile(const std::shared_ptr<FileLocation> &location)

Delete a file at the storage service (incurs simulated overheads)

Parameters:

location – a location

void deleteFile(S4U_CommPort *answer_commport, const std::shared_ptr<FileLocation> &location, bool wait_for_answer)

Delete a file on the storage service.

Parameters:

  • answer_commport – the answer commport to which the reply from the server should be sent

  • location – the location to delete

  • wait_for_answer – whether this call should

virtual void deleteFile(S4U_CommPort *answer_commport, const std::shared_ptr<FileLocation> &location, bool wait_for_answer) override

Delete a file on the storage service.

Parameters:

  • answer_commport – the answer commport to which the reply from the server should be sent

  • location – the location to delete

  • wait_for_answer – whether this call should

std::map<std::string, std::vector<std::shared_ptr<wrench::StorageService>>> &getAllServices()

Method to return the collection of known StorageServices.

Return the set of all services accessible through this CompoundStorageService.

Returns:

The set of known StorageServices.

inline virtual sg_size_t getBufferSize() const override

Determine the storage service’s buffer size.

Returns:

a size in bytes

virtual double getFileLastWriteDate(const std::shared_ptr<FileLocation> &location) override

Get a file’s last write date at a location (in zero simulated time)

Parameters:

location – the location

Returns:

a date in seconds, or -1 if the file is not found

virtual double getLoad() override

Get the load (number of concurrent reads) on the storage service Not implemented yet for CompoundStorageService (is it needed?)

Returns:

the load on the service (currently throws)

inline virtual std::string getMountPoint() override

Return the storage service’s default mountpoint, if any. If none, throws an std::runtime_error exception.

virtual sg_size_t getTotalFreeSpaceAtPath(const std::string &path) override

Synchronously asks the storage services inside the compound storage service for their free space at all of their mount points.

Parameters:

path – a path

Returns:

The free space in bytes at the path

virtual sg_size_t getTotalSpace() override

Get the total space across all internal services known by the CompoundStorageService.

Parameters:

path – the path

Returns:

A number of bytes

inline bool hasFile(const std::shared_ptr<DataFile> &file)

Determines whether a file is present at the storage service (in zero simulated time)

Parameters:

file – a file

Returns:

true if the file is present, false otherwise

inline bool hasFile(const std::shared_ptr<DataFile> &file, const std::string &path)

Determines whether a file is present at the storage service (in zero simulated time)

Parameters:

  • file – a file

  • path – a path

Returns:

true if the file is present, false otherwise

virtual bool hasFile(const std::shared_ptr<FileLocation> &location) override

Check (outside of simulation time) whether the storage service has a file.

Parameters:

location – a location

Returns:

true if the file is present, false otherwise

bool hasFile(const std::shared_ptr<FileLocation> &location) = 0

Determines whether a file is present at the storage service (in zero simulated time)

Parameters:

location – a location

Returns:

true if the file is present, false otherwise

inline virtual bool isBufferized() const override

Determine whether the storage service is bufferized.

Returns:

true if bufferized, false otherwise

inline bool lookupFile(const std::shared_ptr<DataFile> &file)

Lookup whether a file exists on the storage service (incurs simulated overheads)

Parameters:

file – a file

Returns:

true if the file is present, or false

inline bool lookupFile(const std::shared_ptr<DataFile> &file, const std::string &path)

Lookup whether a file exists on the storage service (incurs simulated overheads)

Parameters:

  • file – a file

  • path – a path

Returns:

true if the file is present, or false

inline bool lookupFile(const std::shared_ptr<FileLocation> &location)

Lookup whether a file exists at a location on the storage service (incurs simulated overheads)

Parameters:

location – a location

Returns:

true if the file is present, or false

bool lookupFile(S4U_CommPort *answer_commport, const std::shared_ptr<FileLocation> &location)

Asks the storage service whether it holds a file.

Parameters:

  • answer_commport – the answer commport to which the reply from the server should be sent

  • location – the location to lookup

Returns:

true if the file is present, false otherwise

virtual bool lookupFile(S4U_CommPort *answer_commport, const std::shared_ptr<FileLocation> &location) override

Asks the storage service whether it holds a file.

Parameters:

  • answer_commport – the answer commport to which the reply from the server should be sent

  • location – the location to lookup

Returns:

true if the file is present, false otherwise

std::vector<std::shared_ptr<FileLocation>> lookupFileLocation(const std::shared_ptr<DataFile> &file, S4U_CommPort *answer_commport)

Lookup for a DataFile in the internal file mapping of the CompoundStorageService (a simplified FileRegistry)

Parameters:

  • file – the file of interest

  • answer_commport – the answer commport to which the reply from the server should be sent

Returns:

A vector of shared_ptr on a FileLocation if the DataFile is known to the CompoundStorageService or empty vector if it’s not.

std::vector<std::shared_ptr<FileLocation>> lookupFileLocation(const std::shared_ptr<FileLocation> &location)

Lookup for a FileLocation (using its internal DataFile) in the internal file mapping of the CompoundStorageService (a simplified FileRegistry)

Parameters:

location – the location of interest

Returns:

A shared_ptr on a FileLocation if the DataFile is known to the CompoundStorageService or nullptr if it’s not.

std::vector<std::shared_ptr<FileLocation>> lookupOrDesignateStorageService(const std::shared_ptr<FileLocation> &location)

Lookup for a FileLocation (using its internal DataFile) in the internal file mapping of the CompoundStorageService, and if it is not found, try to allocate the file on one of the underlying storage services, using the user-provided ‘storage_selection’ callback.

Parameters:

location – the location of interest

Returns:

A shared_ptr on a FileLocation if the DataFile is known to the CompoundStorageService or could be allocated or nullptr if it’s not.

std::vector<std::shared_ptr<FileLocation>> lookupOrDesignateStorageService(const std::shared_ptr<FileLocation> &location, unsigned int stripe_count)

Lookup for a FileLocation (using its internal DataFile) in the internal file mapping of the CompoundStorageService, and if it is not found, try to allocate the file on one of the underlying storage services, using the user-provided ‘storage_selection’ callback.

Parameters:

  • location – the location of interest

  • stripe_count – number of stripes required for the given file (overrides any global setting)

Returns:

A shared_ptr on a FileLocation if the DataFile is known to the CompoundStorageService or could be allocated or nullptr if it’s not.

inline void readFile(const std::shared_ptr<DataFile> &file)

Read a file at the storage service (incurs simulated overheads)

Parameters:

file – a file

inline void readFile(const std::shared_ptr<DataFile> &file, const std::string &path)

Read a file at the storage service (incurs simulated overheads)

Parameters:

  • file – a file

  • path – a path

inline void readFile(const std::shared_ptr<DataFile> &file, const std::string &path, sg_size_t num_bytes)

Read a file at the storage service (incurs simulated overheads)

Parameters:

  • file – a file

  • path – a path

  • num_bytes – a number of bytes to read

inline void readFile(const std::shared_ptr<DataFile> &file, sg_size_t num_bytes)

Read a file at the storage service (incurs simulated overheads)

Parameters:

  • file – a file

  • num_bytes – a number of bytes to read

inline void readFile(const std::shared_ptr<FileLocation> &location)

Read a file at the storage service (incurs simulated overheads)

Parameters:

location – a location

inline void readFile(const std::shared_ptr<FileLocation> &location, sg_size_t num_bytes)

Read a file at the storage service (incurs simulated overheads)

Parameters:

  • location – a location

  • num_bytes – a number of bytes to read

void readFile(S4U_CommPort *answer_commport, const std::shared_ptr<FileLocation> &location, sg_size_t num_bytes, bool wait_for_answer)

Read a file from the storage service.

Parameters:

  • answer_commport – the commport on which to expect the answer

  • location – the location

  • num_bytes – the number of bytes to read

  • wait_for_answer – whether to wait for the answer

virtual void readFile(S4U_CommPort *answer_commport, const std::shared_ptr<FileLocation> &location, sg_size_t num_bytes, bool wait_for_answer) override

Read a file from the storage service.

Parameters:

  • answer_commport – the answer commport to which the reply from the server should be sent

  • location – the location

  • num_bytes – the number of bytes to read

  • wait_for_answer – whether to wait for the answer

inline virtual void removeDirectory(const std::string &path) override

Remove a directory and all its content at the storage service (in zero simulated time)

Parameters:

path – a path

inline virtual void removeFile(const std::shared_ptr<FileLocation> &location) override

Remove a file at the storage service (in zero simulated time)

Parameters:

location – a location

inline virtual bool reserveSpace(std::shared_ptr<FileLocation> &location) override

Reserve space at the storage service.

Parameters:

location – a location

Returns:

true if success, false otherwise

virtual void setIsScratch(bool is_scratch) override

setIsScratch can’t be used on a CompoundStorageService because it doesn’t have any actual storage resources.

Parameters:

is_scratch – true or false

inline virtual void unreserveSpace(std::shared_ptr<FileLocation> &location) override

Unreserve space at the storage service.

Parameters:

location – a location

inline void writeFile(const std::shared_ptr<DataFile> &file)

Write a file at the storage service (incurs simulated overheads)

Parameters:

file – a file

inline void writeFile(const std::shared_ptr<DataFile> &file, const std::string &path)

Write a file at the storage service (incurs simulated overheads)

Parameters:

  • file – a file

  • path – a path

inline void writeFile(const std::shared_ptr<FileLocation> &location)

Write a file at the storage service (incurs simulated overheads)

Parameters:

location – a location

void writeFile(S4U_CommPort *answer_commport, const std::shared_ptr<FileLocation> &location, sg_size_t num_bytes_to_write, bool wait_for_answer)

Synchronously write a file to the storage service.

Parameters:

  • answer_commport – the commport on which to expect the answer

  • location – the location

  • num_bytes_to_write – the number of bytes to write to the file.

  • wait_for_answer – whether to wait for the answer

virtual void writeFile(S4U_CommPort *answer_commport, const std::shared_ptr<FileLocation> &location, sg_size_t num_bytes_to_write, bool wait_for_answer) override

Synchronously write a file to the storage service.

Parameters:

  • answer_commport – the answer commport to which the reply from the server should be sent

  • location – the location

  • num_bytes_to_write – number of bytes to write to the file

  • wait_for_answer – whether to wait for the answer

Public Members

std::map<std::string, AllocationTrace> copy_traces = {}

File copy traces.

std::map<std::string, AllocationTrace> delete_traces = {}

File delete traces.

std::vector<std::pair<double, AllocationTrace>> internal_storage_use = {}

Internal storage use.

std::map<std::string, AllocationTrace> write_traces = {}

File read traces.

File write traces

Public Static Functions

static void copyFile(const std::shared_ptr<FileLocation> &src_location, const std::shared_ptr<FileLocation> &dst_location)

Intended to be called by StorageService::copyFile() when the use of a CSS is detected in a file copy.

Synchronously copy a file.

Parameters:

  • src_location – the source location

  • dst_location – the destination location