[−][src]Struct shardio::ShardWriter
Write a stream data items of type T
to disk, in the sort order defined by S
.
Data is buffered up to item_buffer_size
items, then sorted, block compressed and written to disk.
When the ShardWriter is dropped or has finish()
called on it, it will
flush remaining items to disk, write an index of the chunk data and close the file.
The get_sender()
methods returns a ShardSender
that must be used to send items to the writer.
You must close each ShardSender by calling its finish()
method, or data may be lost.
Sorting
Items are sorted according to the Ord
implementation of type S::Key
. Type S
, implementing the SortKey
trait
maps items of type T
to their sort key of type S::Key
. By default the sort key is the data item itself, and the
the DefaultSort
implementation of SortKey
is the identity function.
Methods
impl<T, S> ShardWriter<T, S> where
T: 'static + Send + Serialize,
S: SortKey<T>,
<S as SortKey<T>>::Key: 'static + Send + Ord + Serialize + Clone,
[src]
T: 'static + Send + Serialize,
S: SortKey<T>,
<S as SortKey<T>>::Key: 'static + Send + Ord + Serialize + Clone,
pub fn new<P: AsRef<Path>>(
path: P,
sender_buffer_size: usize,
disk_chunk_size: usize,
item_buffer_size: usize
) -> Result<ShardWriter<T, S>, Error>
[src]
path: P,
sender_buffer_size: usize,
disk_chunk_size: usize,
item_buffer_size: usize
) -> Result<ShardWriter<T, S>, Error>
Create a writer for storing data items of type T
.
Arguments
path
- Path to newly created output filesender_buffer_size
- number of items to buffer on the sending thread before transferring data to the writerdisk_chunk_size
- number of items to store in chunk on disk. Controls the tradeoff between indexing overhead and the granularity can be read at.item_buffer_size
- number of items to buffer before writing data to disk. More buffering causes the data for a given interval to be spread over fewer disk chunks, but requires more memory.
pub fn get_sender(&self) -> ShardSender<T>
[src]
Get a ShardSender
. It can be sent to another thread that is generating data.
pub fn finish(&mut self) -> Result<usize, Error>
[src]
Call finish if you want to detect errors in the writer IO.
Trait Implementations
impl<T, S> Drop for ShardWriter<T, S>
[src]
Auto Trait Implementations
impl<T, S> Send for ShardWriter<T, S> where
S: Send,
T: Send,
S: Send,
T: Send,
impl<T, S> Sync for ShardWriter<T, S> where
S: Sync,
T: Send,
S: Sync,
T: Send,
Blanket Implementations
impl<T> From for T
[src]
impl<T, U> Into for T where
U: From<T>,
[src]
U: From<T>,
impl<T, U> TryFrom for T where
U: Into<T>,
[src]
U: Into<T>,
type Error = Infallible
The type returned in the event of a conversion error.
fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>
[src]
impl<T> Borrow for T where
T: ?Sized,
[src]
T: ?Sized,
impl<T> Any for T where
T: 'static + ?Sized,
[src]
T: 'static + ?Sized,
impl<T> BorrowMut for T where
T: ?Sized,
[src]
T: ?Sized,
fn borrow_mut(&mut self) -> &mut T
[src]
impl<T, U> TryInto for T where
U: TryFrom<T>,
[src]
U: TryFrom<T>,