LuceneRDD

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
lazy val Config: Config

Definition Classes
Configurable
val IndexAnalyzerConfigName: Option[String]

Attributes
protected
Definition Classes
AnalyzerConfigurable
val LuceneSimilarityConfigValue: Option[String]

Attributes
protected
Definition Classes
SimilarityConfigurable
val QueryAnalyzerConfigName: Option[String]

Attributes
protected
Definition Classes
AnalyzerConfigurable
def apply(dataFrame: DataFrame): LuceneRDD[Row]

Constructor with default index, query analyzers and Lucene similarity
Constructor with default index, query analyzers and Lucene similarity
dataFrame
Input DataFrame
def apply(dataFrame: DataFrame, indexAnalyzer: String, queryAnalyzer: String, similarity: String): LuceneRDD[Row]

Instantiate a LuceneRDD with DataFrame
Instantiate a LuceneRDD with DataFrame
dataFrame
Spark DataFrame
indexAnalyzer
Index Analyzer name
queryAnalyzer
Query Analyzer name
similarity
Lucene scoring similarity, i.e., BM25 or TF-IDF
def apply[T](elems: Iterable[T])(implicit arg0: ClassTag[T], sc: SparkContext, conv: (T) ⇒ Document): LuceneRDD[T]
def apply[T](elems: Iterable[T], indexAnalyzer: String, queryAnalyzer: String, similarity: String)(implicit arg0: ClassTag[T], sc: SparkContext, conv: (T) ⇒ Document): LuceneRDD[T]

Instantiate a LuceneRDD with an iterable
Instantiate a LuceneRDD with an iterable
T
Input type
elems
Elements to index
indexAnalyzer
Index Analyzer name
queryAnalyzer
Query Analyzer name
similarity
Lucene scoring similarity, i.e., BM25 or TF-IDF
sc
Spark Context
def apply[T](elems: RDD[T])(implicit arg0: ClassTag[T], conv: (T) ⇒ Document): LuceneRDD[T]
def apply[T](elems: RDD[T], indexAnalyzer: String, queryAnalyzer: String, similarity: String)(implicit arg0: ClassTag[T], conv: (T) ⇒ Document): LuceneRDD[T]

Instantiate a LuceneRDD given an RDD[T]
Instantiate a LuceneRDD given an RDD[T]
T
Generic type
elems
RDD of type T
indexAnalyzer
Index Analyzer name
queryAnalyzer
Query Analyzer name
similarity
Lucene scoring similarity, i.e., BM25 or TF-IDF
final def asInstanceOf[T0]: T0

Definition Classes
Any
def blockDedup(entities: DataFrame, rowToQueryString: (Row) ⇒ String, blockingColumns: Array[String], topK: Int = 3, indexAnalyzer: String = ..., queryAnalyzer: String = ..., similarity: String = getOrElseClassic()): RDD[(Row, Array[SparkScoreDoc])]

Deduplication via blocking
Deduplication via blocking
entities
Entities DataFrame to deduplicate
rowToQueryString
Function that maps Row to Lucene Query String
blockingColumns
Columns on which exact match is required
topK
Number of top-K query results
indexAnalyzer
Lucene analyzer at index time
queryAnalyzer
Lucene analyzer at query time
similarity
Lucene Similarity metric (BM25, Tf/idf)
def blockEntityLinkage(queries: DataFrame, entities: DataFrame, rowToQueryString: (Row) ⇒ String, queryPartColumns: Array[String], entityPartColumns: Array[String], topK: Int = 3, indexAnalyzer: String = ..., queryAnalyzer: String = ..., similarity: String = getOrElseClassic()): RDD[(Row, Array[SparkScoreDoc])]

Entity linkage between two DataFrame by blocking / filtering on one or more columns.
Entity linkage between two DataFrame by blocking / filtering on one or more columns.
queries
Queries / entities to be linked with @corpus
entities
DataFrame of entities to be linked with queries parameter
rowToQueryString
Converts each Row to a 'Lucene Query Syntax'
queryPartColumns
List of query columns for HashPartitioner
entityPartColumns
List of entity columns for HashPartitioner
topK
Number of linked results
indexAnalyzer
Lucene analyzer at index time
queryAnalyzer
Lucene analyzer at query time
similarity
Lucene Similarity metric (BM25, Tf/idf)
returns
Returns top-k linked results as RDD of Tuple2 where _1 is query and _2 is top-k linked results as SparkScoreDoc.
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
def getAnalyzer(analyzerName: Option[String]): Analyzer

Attributes
protected
Definition Classes
AnalyzerConfigurable
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def getOrElseClassic(): String

Attributes
protected
Definition Classes
SimilarityConfigurable
def getOrElseEn(analyzerName: Option[String]): String

Get the configured analyzers or fallback to English
Get the configured analyzers or fallback to English

Attributes
protected
Definition Classes
AnalyzerConfigurable
def getSimilarity(similarityName: Option[String]): Similarity

Attributes
protected
Definition Classes
SimilarityConfigurable
def hashCode(): Int

Definition Classes
AnyRef → Any
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toString(): String

Definition Classes
AnyRef → Any
def version(): Map[String, Any]

Return project information, i.e., version number, build time etc
Return project information, i.e., version number, build time etc

Definition Classes
Versionable
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Docs: class LuceneRDD | package lucenerdd

object LuceneRDD extends Versionable with AnalyzerConfigurable with SimilarityConfigurable

Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

lazy val Config: Config

val IndexAnalyzerConfigName: Option[String]

val LuceneSimilarityConfigValue: Option[String]

val QueryAnalyzerConfigName: Option[String]

def apply(dataFrame: DataFrame): LuceneRDD[Row]

def apply(dataFrame: DataFrame, indexAnalyzer: String, queryAnalyzer: String, similarity: String): LuceneRDD[Row]

def apply[T](elems: Iterable[T])(implicit arg0: ClassTag[T], sc: SparkContext, conv: (T) ⇒ Document): LuceneRDD[T]

def apply[T](elems: Iterable[T], indexAnalyzer: String, queryAnalyzer: String, similarity: String)(implicit arg0: ClassTag[T], sc: SparkContext, conv: (T) ⇒ Document): LuceneRDD[T]

def apply[T](elems: RDD[T])(implicit arg0: ClassTag[T], conv: (T) ⇒ Document): LuceneRDD[T]

def apply[T](elems: RDD[T], indexAnalyzer: String, queryAnalyzer: String, similarity: String)(implicit arg0: ClassTag[T], conv: (T) ⇒ Document): LuceneRDD[T]

final def asInstanceOf[T0]: T0

def blockDedup(entities: DataFrame, rowToQueryString: (Row) ⇒ String, blockingColumns: Array[String], topK: Int = 3, indexAnalyzer: String = ..., queryAnalyzer: String = ..., similarity: String = getOrElseClassic()): RDD[(Row, Array[SparkScoreDoc])]

def clone(): AnyRef

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def finalize(): Unit

def getAnalyzer(analyzerName: Option[String]): Analyzer

final def getClass(): Class[_]

def getOrElseClassic(): String

def getOrElseEn(analyzerName: Option[String]): String

def getSimilarity(similarityName: Option[String]): Similarity

def hashCode(): Int

final def isInstanceOf[T0]: Boolean

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

final def synchronized[T0](arg0: ⇒ T0): T0

def toString(): String

def version(): Map[String, Any]

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from SimilarityConfigurable

Inherited from AnalyzerConfigurable

Inherited from Configurable

Inherited from Serializable

Inherited from Serializable

Inherited from Versionable

Inherited from AnyRef

Inherited from Any

Ungrouped