NewsgroupsIndexer

Example application to index each article in the 20 newsgroups data as a Solr document. The 20 newsgroups data is downloadable from http://qwone.com/~jason/20Newsgroups/.

Articles in any of the three archives available there can be indexed, after first downloading it from the above page and unpacking it.

The path you supply as an argument to the --path cmdline option (see the "Example invocation" section below) should be the directory containing the newsgroup folders. All files found recursively under this path will be indexed to Solr.

Prerequisites

Start Solr in cloud mode, and create a target collection, e.g. (after downloading the binary Solr distribution - see https://lucene.apache.org/solr/ - then unpacking and changing to the unpacked root directory, e.g. solr-5.4.1/):

bin/solr -c && bin/solr create -c testing -shards 2

Document fields

Each header present in the newsgroup articles will be indexed to a Solr dynamic field name prefixed with the header name, e.g. Subject: text will be indexed into a field named Subject_txt_en.

Note that the set of headers in each of the three available archives is different; details are on the download page above.

The body of each article will be indexed into the content_txt_en field.

The newsgroup_s field will contain the name of the article's parent directory.

The id field value will be in the format newsgroup_articlenum, e.g. "comp.graphics_38659", where newsgroup is the name of the article's parent directory, and articlenum is the article filename.

The filepath_s field will contain the full path of the article source file.

If you downloaded the 20news-19997.tar.gz archive, the only one with the Date: header, dates will be indexed into two fields: the Date_s field will contain the original Date: text, and the Date_tdt field will contain the date reformatted in ISO-8601 format.

Example invocation

You must first run mvn -DskipTests package in the spark-solr project, and you must download a Spark 1.6.1 binary distribution and point the environment variable $SPARK_HOME to the unpacked distribution directory.

$SPARK_HOME/bin/spark-submit --master 'local[2]' --class com.lucidworks.spark.SparkApp \
target/spark-solr-2.0.0-SNAPSHOT-shaded.jar newsgroups2solr -zkHost localhost:9983     \
-collection ml20news -path /relative/or/absolute/path/to/20news-18828`

To see a description of all available options, run the following:

$SPARK_HOME/bin/spark-submit --class com.lucidworks.spark.SparkApp \
target/spark-solr-2.0.0-SNAPSHOT-shaded.jar newsgroups2solr --help

Linear Supertypes

LazyLogging, RDDProcessor, Serializable, AnyRef, Any

Instance Constructors

new NewsgroupsIndexer()

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def asInstanceOf[T0]: T0

Definition Classes
Any
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def getName(): String

Definition Classes
NewsgroupsIndexer → RDDProcessor
def getOptions(): Array[Option]

Definition Classes
NewsgroupsIndexer → RDDProcessor
def hashCode(): Int

Definition Classes
AnyRef → Any
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
def loadNewsgroupArticle(stream: PortableDataStream): SolrInputDocument
lazy val logger: Logger

Attributes
protected
Definition Classes
LazyLogging
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
def parseFilePath(basePath: String, filePath: String): (Option[String], String)
def run(conf: SparkConf, cli: CommandLine): Int

Definition Classes
NewsgroupsIndexer → RDDProcessor
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toString(): String

Definition Classes
AnyRef → Any
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Docs: object NewsgroupsIndexer | package ml

class NewsgroupsIndexer extends RDDProcessor with LazyLogging

Prerequisites

Document fields

Example invocation

Instance Constructors

new NewsgroupsIndexer()

Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

final def asInstanceOf[T0]: T0

def clone(): AnyRef

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def finalize(): Unit

final def getClass(): Class[_]

def getName(): String

def getOptions(): Array[Option]

def hashCode(): Int

final def isInstanceOf[T0]: Boolean

def loadNewsgroupArticle(stream: PortableDataStream): SolrInputDocument

lazy val logger: Logger

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

def parseFilePath(basePath: String, filePath: String): (Option[String], String)

def run(conf: SparkConf, cli: CommandLine): Int

final def synchronized[T0](arg0: ⇒ T0): T0

def toString(): String

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from LazyLogging

Inherited from RDDProcessor

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped