pct_tools.io¶

Reading and writing sparse matrices and vectors in the package’s binary formats.

Matrices are 32-bit float sparse matrices in compressed sparse row (CSR) form with 32-bit indices; vectors are arrays of 32-bit floats. All integers below are little-endian 32-bit signed, as written by the C++ code on the platforms the wheels are built for.

Uncompressed matrix (read_matrix):

rows, cols, nnz, outer_size          4 x int32
values                               nnz x float32
outer indices (row pointers)         outer_size x int32
inner indices (column indices)       nnz x int32

Uncompressed vector (store_vector / read_vector):

size                                 int32
values                               size x float32

Compressed matrix (store_compressed_matrix / read_compressed_matrix):

rows, cols, nnz                      3 x int32
sizes of the three chunks below      3 x int32
values chunk                         Blosc2 frame of nnz x float32
outer index chunk                    Blosc2 frame of (rows + 1) x int32
inner index chunk                    Blosc2 frame of nnz x int32

Compressed vector (store_compressed_vector / read_compressed_vector):

size, chunk size                     2 x int32
values chunk                         Blosc2 frame of size x float32

Each Blosc2 chunk records its own codec and shuffle filter, so files written with any codec or shuffle method can be read back without specifying them.

class pct_tools.io.ShuffleMethod(*values)[source]¶

Blosc2 pre-compression filter applied to the raw bytes.

Shuffling rearranges the bytes (or bits) of each element so that similar bytes become contiguous, which usually improves the compression ratio for numerical data.

BLOSC_NOSHUFFLE = 0¶: Compress the bytes as they are.

BLOSC_SHUFFLE = 1¶

fast, with a moderate gain in ratio.

Type:: Byte shuffling

BLOSC_BITSHUFFLE = 2¶

slower, usually the best ratio.

Type:: Bit shuffling

pct_tools.io.DEFAULT_CODEC: Codec = 'lz4hc'¶

a good ratio with fast decompression.

Type:: Codec used when none is given

pct_tools.io.construct_matrix(filename, row_indexes, col_indexes, values, shape, compression_level=9, shuffle_method=ShuffleMethod.BLOSC_SHUFFLE, codec='lz4hc', verbose_level=0, n_threads=4)[source]¶

Construct a sparse matrix from (row, column, value) triplets and store it compressed.

Duplicate (row, column) pairs are summed. The shape is given explicitly rather than inferred from the indices so that trailing empty rows or columns are preserved.

Parameters:

filename (str) – The filename of the output file.
row_indexes (NDArray[np.int32]) – The row indices of the matrix elements, an int32 array of shape (n_elements,).
col_indexes (NDArray[np.int32]) – The column indices of the matrix elements, an int32 array of shape (n_elements,).
values (NDArray[np.float32]) – The values of the matrix elements, a float32 array of shape (n_elements,).
shape (tuple[int, int]) – The (n_rows, n_cols) shape of the resulting matrix.
compression_level (int) – The compression level 0-9 to use. Default is 9.
shuffle_method (ShuffleMethod) – The blosc shuffle method to use. Default is byte shuffling.
codec (Codec) – The Blosc codec to use. Default is LZ4HC.
verbose_level (int) – The verbosity level.
n_threads (int) – The number of threads to use.

Raises:

TypeError – If the index arrays are not int32 or values is not float32.

pct_tools.io.read_matrix(filename)[source]¶

Read an uncompressed Eigen matrix from disk.

Parameters:: filename (str) – Name of the file containing the matrix.
Returns:: The matrix as a sparse matrix of 32-bit floats in CSR format.
Return type:: csr_matrix

pct_tools.io.read_compressed_matrix(filename, verbosity_level=0)[source]¶

Read compressed Eigen matrix from disk.

Parameters:

filename (str) – Name of the file containing the matrix.
verbosity_level (int) – The verbosity level.

Returns:

The matrix as a sparse matrix of 32-bit floats in CSR format.

Return type:

csr_matrix

pct_tools.io.store_compressed_matrix(matrix, filename, compression_level=9, shuffle_method=ShuffleMethod.BLOSC_SHUFFLE, codec='lz4hc', verbosity_level=0)[source]¶

Compress and store a sparse matrix.

Parameters:

matrix (csr_matrix) – The matrix to store, as a sparse matrix of 32-bit floats in CSR format.
filename (str) – The filename of the output file.
compression_level (int) – The compression level 0-9 to use. Default is 9.
shuffle_method (ShuffleMethod) – The blosc shuffle method to use. Default is byte shuffling.
codec (Codec) – The Blosc codec to use. Default is LZ4HC.
verbosity_level (int) – The verbosity level.

pct_tools.io.read_compressed_vector(filename)[source]¶

Read compressed Eigen vector from disk.

Parameters:: filename (str) – Name of the file containing the vector.
Returns:: The vector as a one-dimensional array of 32-bit floats.
Return type:: NDArray[np.float32]

pct_tools.io.read_vector(filename)[source]¶

Read an Eigen vector from disk.

Parameters:: filename (str) – Name of the file containing the vector.
Returns:: The vector as a one-dimensional array of 32-bit floats.
Return type:: NDArray[np.float32]

pct_tools.io.store_compressed_vector(vector, filename, compression_level=9, shuffle_method=ShuffleMethod.BLOSC_SHUFFLE, codec='lz4hc')[source]¶

Compress and store a vector.

Parameters:

vector (NDArray[np.float32]) – The one-dimensional float32 array to compress and store.
filename (str) – The filename of the output file.
compression_level (int) – The compression level to use.
shuffle_method (ShuffleMethod) – The blosc shuffle method to use. Default is byte shuffling.
codec (Codec) – The Blosc codec to use. Default is LZ4HC.

Raises:

TypeError – If vector is not a float32 array.

pct_tools.io.store_vector(vector, filename)[source]¶

Store a vector.

Parameters:

vector (NDArray[np.float32]) – The one-dimensional float32 array to store.
filename (str) – The filename of the output file.

Raises:

TypeError – If vector is not a float32 array.