water.fvec

Class Frame

java.lang.Object

water.Iced<T>

water.Keyed<T>

water.Lockable<Frame>

water.fvec.Frame

All Implemented Interfaces:

java.io.Externalizable, java.io.Serializable, java.lang.Cloneable, Freezable<Frame>

public class Frame
extends Lockable<Frame>

A collection of named Vecs, essentially an R-like Distributed Data Frame.

Frames represent a large distributed 2-D table with named columns (Vecs) and numbered rows. A reasonable column limit is 100K columns, but there's no hard-coded limit. There's no real row limit except memory; Frames (and Vecs) with many billions of rows are used routinely.

A Frame is a collection of named Vecs; a Vec is a collection of numbered Chunks. A Frame is small, cheaply and easily manipulated, it is commonly passed-by-Value. It exists on one node, and may be stored in the DKV. Vecs, on the other hand, must be stored in the DKV, as they represent the shared common management state for a collection of distributed Chunks.

Multiple Frames can reference the same Vecs, although this sharing can make Vec lifetime management complex. Commonly temporary Frames are used to work with a subset of some other Frame (often during algorithm execution, when some columns are dropped from the modeling process). The temporary Frame can simply be ignored, allowing the normal GC process to reclaim it. Such temp Frames usually have a null key.

All the Vecs in a Frame belong to the same Vec.VectorGroup which then enforces Chunk row alignment across Vecs (or at least enforces a low-cost access model). Parallel and distributed execution touching all the data in a Frame relies on this alignment to get good performance.

Example: Make a Frame from a CSV file:

  File file = ...
  NFSFileVec nfs = NFSFileVec.make(file); // NFS-backed Vec, lazily read on demand
  Frame fr = water.parser.ParseDataset.parse(Key.make("myKey"),nfs._key);
  

Example: Find and remove the Vec called "unique_id" from the Frame, since modeling with a unique_id can lead to overfitting:

  Vec uid = fr.remove("unique_id");
  

Example: Move the response column to the last position:

  fr.add("response",fr.remove("response"));
  

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Frame.VecSpecifier Pair of (column name, Frame key).

Field Summary

Fields

Modifier and Type	Field and Description
java.lang.String[]	_names Vec names

Fields inherited from class water.Lockable

_lockers

Fields inherited from class water.Keyed

_key

Constructor Summary

Constructors

Constructor and Description
Frame(Frame fr) Deep copy of Vecs and Keys and Names (but not data!) to a new random Key.
Frame(Key key) Creates an empty frame with given key.
Frame(Key key, java.lang.String[] names, Vec[] vecs) Creates a frame with given key, names and vectors.
Frame(Key key, Vec[] vecs, boolean noChecks) Special constructor for data with unnamed columns (e.g.
Frame(java.lang.String[] names, Vec[] vecs) Creates an internal frame composed of the given Vecs and names.
Frame(Vec... vecs) Creates an internal frame composed of the given Vecs and default names.

Method Summary

Methods

Modifier and Type	Method and Description
Frame	add(Frame fr) Append a Frame onto this Frame.
void	add(java.lang.String[] names, Vec[] vecs)
Vec	add(java.lang.String name, Vec vec) Append a named Vec to the Frame.
Vec	anyVec() Returns the first readable vector.
long	byteSize() The Vec.byteSize of all Vecs
boolean	checkCompatible(Frame fr) Quick compatibility check between Frames.
protected long	checksum_impl() 64-bit checksum of the checksums of the vecs.
Frame	deepSlice(java.lang.Object orows, java.lang.Object ocols) In support of R, a generic Deep Copy and Slice.
static java.lang.String	defaultColName(int col) Default column name maker
java.lang.String[][]	domains() All the domains for enum columns; null for non-enum columns.
Frame	extractFrame(int startIdx, int endIdx) Split this Frame; return a subframe created from the given column interval, and remove those columns from this Frame.
int	find(java.lang.String name) Finds the column index with a matching name, or -1 if missing
int[]	find(java.lang.String[] names) Bulk find(String) api
int	find(Vec vec) Finds the matching column index, or -1 if missing
Key[]	keys() The array of keys.
Vec	lastVec() Convenience to accessor for last Vec
java.lang.String	lastVecName() Convenience to accessor for last Vec name
Frame	makeCompatible(Frame f) Return Frame 'f' if 'f' is compatible with 'this', else return a new Frame compatible with 'this' and a copy of 'f's data otherwise.
java.lang.String	name(int i) A single column name.
java.lang.String[]	names() The array of column names.
int	numCols() Number of columns
long	numRows() Number of rows
Futures	postWrite(Futures fs) Allow rollups for all written-into vecs; used by MRTask once writing is complete.
Vec[]	reloadVecs() Force a cache-flush and reload, assuming vec mappings were altered remotely, or that the _vecs array was shared and now needs to be a defensive copy.
Futures	remove_impl(Futures fs) Actually remove/delete all Vecs from memory, not just from the Frame.
Vec	remove(int idx) Removes a numbered column.
Vec[]	remove(int[] idxs) Removes a list of columns by index; the index list must be sorted
Vec	remove(java.lang.String name) Removes the column with a matching name.
Frame	remove(java.lang.String[] names)
Vec	replace(int col, Vec nv) Replace one column with another.
void	restructure(java.lang.String[] names, Vec[] vecs) Restructure a Frame completely
Frame	subframe(java.lang.String[] names) Returns a subframe of this frame containing only vectors with desired names.
Frame[]	subframe(java.lang.String[] names, double c) Returns a new frame composed of vectors of this frame selected by given names.
void	swap(int lo, int hi) Swap two Vecs in-place; useful for sorting columns by some criteria
java.io.InputStream	toCSV(boolean headers, boolean hex_string) Convert this Frame to a CSV (in an InputStream), that optionally is compatible with R 3.1's recent change to read.csv()'s behavior.
java.lang.String	toString()
java.lang.String	toString(long off, int len)
Vec	vec(int idx) Returns the Vec by given index, implemented by code: vecs()[idx].
Vec	vec(java.lang.String name) Return a Vec by name, or null if missing
Vec[]	vecs() The internal array of Vecs.
Vec[]	vecs(int[] idxs)

Methods inherited from class water.Lockable

delete_and_lock, delete, delete, delete, read_lock, read_lock, unlock_all, unlock, update, write_lock

Methods inherited from class water.Keyed

checksum, remove, remove, remove, remove

Methods inherited from class water.Iced

clone, frozenType, read_impl, read, readExternal, readJSON_impl, readJSON, toJsonString, write_impl, write, writeExternal, writeHTML_impl, writeHTML, writeJSON_impl, writeJSON

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

_names

public java.lang.String[] _names

Vec names

Constructor Detail

Frame

public Frame(Vec... vecs)

Creates an internal frame composed of the given Vecs and default names. The frame has no key.

Frame

public Frame(java.lang.String[] names,
     Vec[] vecs)

Creates an internal frame composed of the given Vecs and names. The frame has no key.

Frame

public Frame(Key key)

Creates an empty frame with given key.

Frame

public Frame(Key key,
     Vec[] vecs,
     boolean noChecks)

Special constructor for data with unnamed columns (e.g. svmlight) bypassing *all* checks.

Parameters:

key -

vecs -

noChecks -

Frame

public Frame(Key key,
     java.lang.String[] names,
     Vec[] vecs)

Creates a frame with given key, names and vectors.

Frame

public Frame(Frame fr)

Deep copy of Vecs and Keys and Names (but not data!) to a new random Key. The resulting Frame does not share with the original, so the set of Vecs can be freely hacked without disturbing the original Frame.

Method Detail

defaultColName

public static java.lang.String defaultColName(int col)

Default column name maker

checkCompatible

public boolean checkCompatible(Frame fr)

Quick compatibility check between Frames. Used by some tests for efficient equality checks.

numCols

public int numCols()

Number of columns

Returns:

Number of columns

numRows

public long numRows()

Number of rows

Returns:

Number of rows

anyVec

public final Vec anyVec()

Returns the first readable vector.

Returns:

the first readable Vec

names

public java.lang.String[] names()

The array of column names.

Returns:

the array of column names

name

public java.lang.String name(int i)

A single column name.

Returns:

the column name

keys

public Key[] keys()

The array of keys.

Returns:

the array of keys for each vec in the frame.

vecs

public final Vec[] vecs()

The internal array of Vecs. For efficiency Frames contain an array of Vec Keys - and the Vecs themselves are lazily loaded from the DKV.

Returns:

the internal array of Vecs

vecs

public final Vec[] vecs(int[] idxs)

lastVec

public Vec lastVec()

Convenience to accessor for last Vec

Returns:

last Vec

lastVecName

public java.lang.String lastVecName()

Convenience to accessor for last Vec name

Returns:

last Vec name

reloadVecs

public final Vec[] reloadVecs()

Force a cache-flush and reload, assuming vec mappings were altered remotely, or that the _vecs array was shared and now needs to be a defensive copy.

Returns:

the new instance of the Frame's Vec[]

vec

public final Vec vec(int idx)

Returns the Vec by given index, implemented by code: vecs()[idx].

Parameters:

idx - idx of column

Returns:

this frame idx-th vector, never returns null

vec

public Vec vec(java.lang.String name)

Return a Vec by name, or null if missing

Returns:

a Vec by name, or null if missing

find

public int find(java.lang.String name)

Finds the column index with a matching name, or -1 if missing

Returns:

the column index with a matching name, or -1 if missing

find

public int find(Vec vec)

Finds the matching column index, or -1 if missing

Returns:

the matching column index, or -1 if missing

find

public int[] find(java.lang.String[] names)

Bulk find(String) api

Returns:

An array of column indices matching the names array

domains

public java.lang.String[][] domains()

All the domains for enum columns; null for non-enum columns.

Returns:

the domains for enum columns

byteSize

public long byteSize()

The Vec.byteSize of all Vecs

Returns:

the Vec.byteSize of all Vecs

checksum_impl

protected long checksum_impl()

64-bit checksum of the checksums of the vecs. SHA-265 checksums of the chunks are XORed together. Since parse always parses the same pieces of files into the same offsets in some chunk this checksum will be consistent across reparses.

Specified by:

checksum_impl in class Keyed<Frame>

Returns:

64-bit Frame checksum

add

public void add(java.lang.String[] names,
       Vec[] vecs)

add

public Vec add(java.lang.String name,
      Vec vec)

Append a named Vec to the Frame. Names are forced unique, by appending a unique number if needed.

Returns:

the added Vec, for flow-coding

add

public Frame add(Frame fr)

Append a Frame onto this Frame. Names are forced unique, by appending unique numbers if needed.

Returns:

the expanded Frame, for flow-coding

swap

public void swap(int lo,
        int hi)

Swap two Vecs in-place; useful for sorting columns by some criteria

subframe

public Frame subframe(java.lang.String[] names)

Returns a subframe of this frame containing only vectors with desired names.

Parameters:

names - list of vector names

Returns:

a new frame which collects vectors from this frame with desired names.

Throws:

java.lang.IllegalArgumentException - if there is no vector with desired name in this frame.

subframe

public Frame[] subframe(java.lang.String[] names,
               double c)

Returns a new frame composed of vectors of this frame selected by given names. The method replaces missing vectors by a constant column filled by given value.

Parameters:

names - names of vector to compose a subframe

c - value to fill missing columns.

Returns:

two frames, the first contains subframe, the second contains newly created constant vectors or null

postWrite

public Futures postWrite(Futures fs)

Allow rollups for all written-into vecs; used by MRTask once writing is complete.

Returns:

the original Futures, for flow-coding

remove_impl

public Futures remove_impl(Futures fs)

Actually remove/delete all Vecs from memory, not just from the Frame.

Overrides:

remove_impl in class Keyed<Frame>

Returns:

the original Futures, for flow-coding

replace

public Vec replace(int col,
          Vec nv)

Replace one column with another. Caller must perform global update (DKV.put) on this updated frame.

Returns:

The old column, for flow-coding

extractFrame

public Frame extractFrame(int startIdx,
                 int endIdx)

Split this Frame; return a subframe created from the given column interval, and remove those columns from this Frame.

Parameters:

startIdx - index of first column (inclusive)

endIdx - index of the last column (exclusive)

Returns:

a new Frame containing specified interval of columns

remove

public Vec remove(java.lang.String name)

Removes the column with a matching name.

Returns:

The removed column

remove

public Frame remove(java.lang.String[] names)

remove

public Vec[] remove(int[] idxs)

Removes a list of columns by index; the index list must be sorted

Returns:

an array of the removed columns

remove

public final Vec remove(int idx)

Removes a numbered column.

Returns:

the removed column

restructure

public void restructure(java.lang.String[] names,
               Vec[] vecs)

Restructure a Frame completely

deepSlice

public Frame deepSlice(java.lang.Object orows,
              java.lang.Object ocols)

In support of R, a generic Deep Copy and Slice.

Semantics are a little odd, to match R's. Each dimension spec can be:

• null - all of them
• a sorted list of negative numbers (no dups) - all BUT these
• an unordered list of positive - just these, allowing dups

The numbering is 1-based; zero's are not allowed in the lists, nor are out-of-range values.

Returns:

the sliced Frame

toString

public java.lang.String toString()

Overrides:

toString in class java.lang.Object

toString

public java.lang.String toString(long off,
                        int len)

makeCompatible

public Frame makeCompatible(Frame f)

Return Frame 'f' if 'f' is compatible with 'this', else return a new Frame compatible with 'this' and a copy of 'f's data otherwise. Note that this can, in the worst case, copy all of thiss' data.

Returns:

This Frame's data in a Frame that is compatible with f.

toCSV

public java.io.InputStream toCSV(boolean headers,
                        boolean hex_string)

Convert this Frame to a CSV (in an InputStream), that optionally is compatible with R 3.1's recent change to read.csv()'s behavior.

Returns:

An InputStream containing this Frame as a CSV