spark
A wrapper around some unserializable objects that make them both Java serializable.
Wraps a value of an unserialized type T in a KryoSerializationWrapper[T], which gives one a way to serialize T.
Wraps a value of an unserialized type T in a KryoSerializationWrapper[T], which gives one a way to serialize T.
NOTE: The vast majority of this code is copied / based off of the classes with the same name in the Apache Shark project.
Original file is here (accessed on April 20, 2015): https://github.com/amplab/shark/blob/master/src/main/scala/shark/execution/serialization/KryoSerializationWrapper.scala
JVM object serialization using Kryo.
JVM object serialization using Kryo. This is much more efficient, but Kryo sometimes is buggy to use. We use this mainly to serialize the object inspectors.
NOTE: The vast majority of this code is copied / based off of the classes with the same name in the Apache Shark project.
Original file is here: https://github.com/amplab/shark/blob/master/src/main/scala/shark/execution/serialization/KryoSerializationWrapper.scala
Methods that either wrap, or operate on wrapped, values so that common RDD operations are available with a natural, functional syntax.
Methods that either wrap, or operate on wrapped, values so that common RDD operations are available with a natural, functional syntax.
Let's look at Map as an example:
// implemented using librray that is not extendable and doesn't implement Serialzable val f: A => B = ... // can be anywhere, error will occur even if in local mode val data: RDD[A] = ... // cannot do data.map(f) // runtime exception :( // as f does not implement Serializable // instead do Map(f)(data) // will serialize it using Kryo and safely // deserialize to perform map on the data RDD
A wrapper around some unserializable objects that make them both Java serializable. Internally, Kryo is used for serialization.
Use KryoSerializationWrapper(value) to create a wrapper.
Note that the value contained in the wrapper is mutable. It must be initialized using Java Serialization (which calls a private readObject method that does the byte-by-byte deserialization).
Also note that this class is both abstract and sealed. The only valid place to create such a wrapper is the companion object's apply method.