Package org.apache.beam.sdk.util

Class BufferedElementCountingOutputStream

  • All Implemented Interfaces:
    java.io.Closeable, java.io.Flushable, java.lang.AutoCloseable
    @Internal
@NotThreadSafe
public class BufferedElementCountingOutputStream
extends java.io.OutputStream
    Provides an efficient encoding for Iterables containing small values by buffering up to bufferSize bytes of data before prefixing the count. Note that each element needs to be encoded in a nested context. See Coder.Context for more details.

    To use this stream: 

    
 BufferedElementCountingOutputStream os = ...
 for (Element E : elements) {
   os.markElementStart();
   // write an element to os
 }
 os.finish();

    The resulting output stream is: 

     countA element(0) element(1) ... element(countA - 1)
 countB element(0) element(1) ... element(countB - 1)
 ...
 countX element(0) element(1) ... element(countX - 1)
 countY

    To read this stream: 

    
 InputStream is = ...
 long count;
 do {
   count = VarInt.decodeLong(is);
   for (int i = 0; i < count; ++i) {
     // read an element from is
   }
 } while(count > 0);

    The counts are encoded as variable length longs. See VarInt.encode(long, OutputStream) for more details. The end of the iterable is detected by reading a count of 0.

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void close()  
      void finish()
      Finishes the encoding by flushing any buffered data, and outputting a final count of 0.
      void flush()  
      void markElementStart()
      Marks that a new element is being output.
      void write​(byte[] b, int off, int len)  
      void write​(int b)  

      • Methods inherited from class java.io.OutputStream

        nullOutputStream, write

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Constructor Detail

      • BufferedElementCountingOutputStream

        public BufferedElementCountingOutputStream​(java.io.OutputStream os)
        Creates an output stream which encodes the number of elements output to it in a streaming manner.

      • BufferedElementCountingOutputStream

        public BufferedElementCountingOutputStream​(java.io.OutputStream os,
                                           long terminatorValue)

    • Method Detail

      • finish

        public void finish()
            throws java.io.IOException
        Finishes the encoding by flushing any buffered data, and outputting a final count of 0.
        Throws:
        java.io.IOException

      • markElementStart

        public void markElementStart()
                      throws java.io.IOException
        Marks that a new element is being output. This allows this output stream to use the buffer if it had previously overflowed marking the start of a new block of elements.
        Throws:
        java.io.IOException

      • write

        public void write​(int b)
           throws java.io.IOException
        Specified by:
        write in class java.io.OutputStream
        Throws:
        java.io.IOException

      • write

        public void write​(byte[] b,
                  int off,
                  int len)
           throws java.io.IOException
        Overrides:
        write in class java.io.OutputStream
        Throws:
        java.io.IOException

      • flush

        public void flush()
           throws java.io.IOException
        Specified by:
        flush in interface java.io.Flushable
        Overrides:
        flush in class java.io.OutputStream
        Throws:
        java.io.IOException

      • close

        public void close()
           throws java.io.IOException
        Specified by:
        close in interface java.lang.AutoCloseable
        Specified by:
        close in interface java.io.Closeable
        Overrides:
        close in class java.io.OutputStream
        Throws:
        java.io.IOException