Storage package#
Submodules#
GCloud Storage module#
- class ablator.modules.storage.cloud.GcpConfig(*args, **kwargs)[source]#
Bases:
ConfigBaseConfiguration for Google Cloud Storage.
- Attributes:
- bucketstr
The bucket to use.
- exclude_globOptional[str]
A glob to exclude from the rsync.
- exclude_chkptsbool
Whether to exclude checkpoints from the rsync.
- __init__(*args, **kwargs)[source]#
Initialize the GcpConfig class for managing Google Cloud Platform configurations.
- Parameters:
- *args
Positional arguments.
- **kwargs
Keyword arguments.
- Raises:
- AssertionError
If the GCP instance is not found.
Notes
The IP address check avoids overly generic hostnames to match with existing instances.
- list_bucket(destination: str | None = None)[source]#
List the contents of a bucket. If destination is None, list the bucket itself.
- Parameters:
- destinationstr | None
Bucket path.
- Returns:
- list[str]
List of files in the bucket.
- rsync_down(remote_path: str, local_path: Path, logger: FileLogger | None = None, verbose=True)[source]#
Rsync files from the bucket.
- Parameters:
- remote_pathstr
The source path in the bucket.
- local_pathPath
The local path to download to.
- loggerFileLogger | None
The logger to use.
- verbosebool
Whether to print the output.
- rsync_down_node(node_hostname, remote_path: str, local_path: Path, logger: FileLogger | None = None, verbose=True)[source]#
Rsync files from the bucket to all nodes with the given hostname.
- Parameters:
- node_hostnamestr
The hostname of the nodes to rsync to.
- remote_pathstr
The source path in the bucket.
- local_pathPath
The local path to download to.
- loggerFileLogger | None
The logger to use.
- verbosebool
Whether to print the output.
- rsync_up(local_path: Path, remote_path: str, logger: FileLogger | None = None)[source]#
Rsync files to the bucket.
- Parameters:
- local_pathPath
The local path to upload.
- remote_pathstr
The destination path in the bucket.
- loggerFileLogger | None
The logger to use.
- Raises:
- AssertionError
If the rsync fails.
Remote Storage module#
- class ablator.modules.storage.remote.RemoteConfig(*args, **kwargs)[source]#
Bases:
ConfigBaseConfiguration for a remote storage.
- Attributes:
- remote_pathstr
The path to the remote storage.
- usernamestr
The username to use for the remote storage.
- hostnamestr
The hostname of the remote storage.
- portNone | int
The port to use for the remote storage.
- exclude_globNone | str
A glob to exclude from the rsync.
- exclude_chkptsbool
Whether to exclude checkpoints from the rsync.
- config_class#
alias of
RemoteConfig
- rsync_down(local_path: Path, destination: str, timeout_s: int | None = None, run_async=False)[source]#
start a new process and download files from the remote storage.
- Parameters:
- local_pathPath
The local path to download to.
- destinationstr
The destination path in the remote storage.
- timeout_sint | None
The timeout in seconds.
- run_asyncbool
Whether to run the command asynchronously.
- rsync_up(local_path: Path, destination: str, timeout_s: int | None = None, run_async=False)[source]#
start a new process and upload files to the remote storage.
- Parameters:
- local_pathPath
The local path to upload.
- destinationstr
The destination path in the remote storage.
- timeout_sint | None
The timeout in seconds.
- run_asyncbool
Whether to run the command asynchronously.
- ablator.modules.storage.remote.run_cmd_wait(cmd, timeout=300, raise_errors=False) str[source]#
Run a command and wait for it to finish. If the command takes longer than
timeoutseconds, kill it. Ifraise_errorsis True, raise aTimeoutExpiredexception.- Parameters:
- cmdstr
The command to run.
- timeoutint
The timeout in seconds.
- raise_errorsbool
Whether to raise errors.
- Returns:
- str
The output of the command.