Struct shardio::ShardReader

pub struct ShardReader<T, S = DefaultSort> where
    S: SortKey<T>,  { /* fields omitted */ }

Read from a collection of shardio files. The input data is merged to give a single sorted view of the combined dataset. The input files must be created with the same sort order S as they are read with.

Methods

impl<T, S> ShardReader<T, S> where
    T: DeserializeOwned,
    <S as SortKey<T>>::Key: Clone + Ord + DeserializeOwned,
    S: SortKey<T>, 

pub fn open<P: AsRef<Path>>(shard_file: P) -> Result<ShardReader<T, S>, Error>

Open a single shard files into reader

pub fn open_set<P: AsRef<Path>>(
    shard_files: &[P]
) -> Result<ShardReader<T, S>, Error>

Open a set of shard files into an aggregated reader

pub fn read_range(
    &self,
    range: &Range<<S as SortKey<T>>::Key>,
    data: &mut Vec<T>
) -> Result<(), Error>

Read data from the given range into data buffer. The data buffer is not cleared before adding items.

pub fn iter_range<'a>(
    &'a self,
    range: &Range<<S as SortKey<T>>::Key>
) -> Result<MergeIterator<'a, T, S>, Error>

Iterate over items in the given range

pub fn iter<'a>(&'a self) -> Result<MergeIterator<'a, T, S>, Error>

Iterate over all items

pub fn len(&self) -> usize

Total number of items

pub fn make_chunks(
    &self,
    num_chunks: usize,
    range: &Range<<S as SortKey<T>>::Key>
) -> Vec<Range<<S as SortKey<T>>::Key>>

Generate num_chunks ranges covering the give range, each with a roughly equal numbers of elements. The ranges can be fed to iter_range