Serialization

All the serialization and deserialization methods used by the TensorProcessor.

Flax

Flax serialization and deserialization utilities.

deserialize(data)

Convert safetensors format to flax tensors using safetensors.flax.load.

Parameters:

Name	Type	Description	Default
data	bytes	Safetensors formatted data.	required

Returns:

Type	Description
Dict[str, Array]	Dict[str, Array]: Flax tensors stored in a dictionary with their name as key.

Source code in src/tensorshare/serialization/flax.py

@require_backend("flax", "flaxlib", "jax")
def deserialize(data: bytes) -> Dict[str, Array]:
    """
    Convert safetensors format to flax tensors using `safetensors.flax.load`.

    Args:
        data (bytes): Safetensors formatted data.

    Returns:
        Dict[str, Array]: Flax tensors stored in a dictionary with their name as key.
    """
    return flax_load(data)  # type: ignore

serialize(tensors, metadata=None)

Convert flax tensors to safetensors format using safetensors.flax.save.

Parameters:

Name	Type	Description	Default
tensors	Dict[str, Array]	Flax tensors stored in a dictionary with their name as key.	required
metadata	Optional[Dict[str, str]]	Metadata to add to the safetensors file. Defaults to None.	None

Returns:

Name	Type	Description
bytes	bytes	Tensors formatted with their metadata if any.

Source code in src/tensorshare/serialization/flax.py

@require_backend("flax", "flaxlib", "jax")
def serialize(
    tensors: Dict[str, Array], metadata: Optional[Dict[str, str]] = None
) -> bytes:
    """
    Convert flax tensors to safetensors format using `safetensors.flax.save`.

    Args:
        tensors (Dict[str, Array]):
            Flax tensors stored in a dictionary with their name as key.
        metadata (Optional[Dict[str, str]], optional):
            Metadata to add to the safetensors file. Defaults to None.

    Returns:
        bytes: Tensors formatted with their metadata if any.
    """
    return flax_save(tensors, metadata=metadata)  # type: ignore

NumPy

NumPy serialization and deserialization utilities.

deserialize(data)

Convert safetensors format to numpy tensors using safetensors.numpy.load.

Parameters:

Name	Type	Description	Default
data	bytes	Safetensors formatted data.	required

Returns:

Type	Description
Dict[str, ndarray]	Dict[str, np.ndarray]: Numpy tensors stored in a dictionary with their name as key.

Source code in src/tensorshare/serialization/numpy.py

@require_backend("numpy")
def deserialize(data: bytes) -> Dict[str, np.ndarray]:
    """
    Convert safetensors format to numpy tensors using `safetensors.numpy.load`.

    Args:
        data (bytes): Safetensors formatted data.

    Returns:
        Dict[str, np.ndarray]: Numpy tensors stored in a dictionary with their name as key.
    """
    return np_load(data)  # type: ignore

serialize(tensors, metadata=None)

Convert numpy tensors to safetensors format using safetensors.numpy.save.

Parameters:

Name	Type	Description	Default
tensors	Dict[str, ndarray]	Numpy tensors stored in a dictionary with their name as key.	required
metadata	Optional[Dict[str, str]]	Metadata to add to the safetensors file. Defaults to None.	None

Returns:

Name	Type	Description
bytes	bytes	Tensors formatted with their metadata if any.

Source code in src/tensorshare/serialization/numpy.py

@require_backend("numpy")
def serialize(
    tensors: Dict[str, np.ndarray], metadata: Optional[Dict[str, str]] = None
) -> bytes:
    """
    Convert numpy tensors to safetensors format using `safetensors.numpy.save`.

    Args:
        tensors (Dict[str, np.ndarray]):
            Numpy tensors stored in a dictionary with their name as key.
        metadata (Optional[Dict[str, str]], optional):
            Metadata to add to the safetensors file. Defaults to None.

    Returns:
        bytes: Tensors formatted with their metadata if any.
    """
    return np_save(tensors, metadata=metadata)  # type: ignore

PaddlePaddle

PaddlePaddle serialization and deserialization utilities

deserialize(data)

Convert safetensors format to paddle tensors using safetensors.paddle.load.

Parameters:

Name	Type	Description	Default
data	bytes	Safetensors formatted data.	required

Returns:

Type	Description
Dict[str, Tensor]	Dict[str, paddle.Tensor]: Paddle tensors stored in a dictionary with their name as key.

Source code in src/tensorshare/serialization/paddle.py

@require_backend("paddle")
def deserialize(data: bytes) -> Dict[str, paddle.Tensor]:
    """
    Convert safetensors format to paddle tensors using `safetensors.paddle.load`.

    Args:
        data (bytes): Safetensors formatted data.

    Returns:
        Dict[str, paddle.Tensor]: Paddle tensors stored in a dictionary with their name as key.
    """
    return paddle_load(data)  # type: ignore

serialize(tensors, metadata=None)

Convert paddle tensors to safetensors format using safetensors.paddle.save.

Parameters:

Name	Type	Description	Default
tensors	Dict[str, Tensor]	Paddle tensors stored in a dictionary with their name as key.	required
metadata	Optional[Dict[str, str]]	Metadata to add to the safetensors file. Defaults to None.	None

Returns:

Name	Type	Description
bytes	bytes	Tensors formatted with their metadata if any.

Source code in src/tensorshare/serialization/paddle.py

@require_backend("paddle")
def serialize(
    tensors: Dict[str, paddle.Tensor], metadata: Optional[Dict[str, str]] = None
) -> bytes:
    """
    Convert paddle tensors to safetensors format using `safetensors.paddle.save`.

    Args:
        tensors (Dict[str, paddle.Tensor]):
            Paddle tensors stored in a dictionary with their name as key.
        metadata (Optional[Dict[str, str]], optional):
            Metadata to add to the safetensors file. Defaults to None.

    Returns:
        bytes: Tensors formatted with their metadata if any.
    """
    return paddle_save(tensors, metadata=metadata)  # type: ignore

PyTorch

Torch serialization and deserialization utilities.

deserialize(data)

Convert safetensors format to torch tensors using safetensors.torch.load.

Parameters:

Name	Type	Description	Default
data	bytes	Safetensors formatted data.	required

Returns:

Type	Description
Dict[str, Tensor]	Dict[str, torch.Tensor]: Torch tensors stored in a dictionary with their name as key.

Source code in src/tensorshare/serialization/torch.py

@require_backend("torch")
def deserialize(data: bytes) -> Dict[str, torch.Tensor]:
    """
    Convert safetensors format to torch tensors using `safetensors.torch.load`.

    Args:
        data (bytes): Safetensors formatted data.

    Returns:
        Dict[str, torch.Tensor]: Torch tensors stored in a dictionary with their name as key.
    """
    return torch_load(data)  # type: ignore

serialize(tensors, metadata=None)

Convert torch tensors to safetensors format using safetensors.torch.save.

Parameters:

Name	Type	Description	Default
tensors	Dict[str, Tensor]	Torch tensors stored in a dictionary with their name as key.	required
metadata	Optional[Dict[str, str]]	Metadata to add to the safetensors file. Defaults to None.	None

Returns:

Name	Type	Description
bytes	bytes	Tensors formatted with their metadata if any.

Source code in src/tensorshare/serialization/torch.py

@require_backend("torch")
def serialize(
    tensors: Dict[str, torch.Tensor], metadata: Optional[Dict[str, str]] = None
) -> bytes:
    """
    Convert torch tensors to safetensors format using `safetensors.torch.save`.

    Args:
        tensors (Dict[str, torch.Tensor]):
            Torch tensors stored in a dictionary with their name as key.
        metadata (Optional[Dict[str, str]], optional):
            Metadata to add to the safetensors file. Defaults to None.

    Returns:
        bytes: Tensors formatted with their metadata if any.
    """
    return torch_save(tensors, metadata=metadata)  # type: ignore

TensorFlow

Tensorflow serialization and deserialization utilities.

---

Last update: 2023-08-20
Created: 2023-08-20