com.netflix.hollow.api.producer

Class HollowProducer

java.lang.Object

com.netflix.hollow.api.producer.HollowProducer

public class HollowProducer
extends java.lang.Object

A HollowProducer is the top-level class used by producers of Hollow data to populate, publish, and announce data states. The interactions between the "blob" store and announcement mechanism are defined by this class, and the implementations of the data publishing and announcing are abstracted in interfaces which are provided to this class.

To obtain a HollowProducer, you should use a builder pattern, for example:

 

 HollowProducer producer = HollowProducer.withPublisher(publisher)
                                         .withAnnouncer(announcer)
                                         .build();
 
 

The following components are injectable, but only an implementation of the HollowProducer.Publisher is required to be injected, all other components are optional. :

HollowProducer.Publisher

Implementations of this class define how to publish blob data to the blob store.

HollowProducer.Announcer

Implementations of this class define the announcement mechanism, which is used to track the version of the currently announced state.

One or more event listeners

Listeners are notified about the progress and status of producer cycles throughout the various cycle stages. Of special note are ValidatorListener that allow for semantic validation of the data contained in a state prior to announcement. If a RuntimeException is thrown during validation, the state will not be announced, and the producer will be automatically rolled back to the prior state.

A Blob staging directory

Before blobs are published, they must be written and inspected/validated. A directory may be specified as a File to which these "staged" blobs will be written prior to publish. Staged blobs will be cleaned up automatically after publish.

HollowProducer.BlobCompressor

Implementations of this class intercept blob input/output streams to allow for compression in the blob store.

HollowProducer.BlobStager

Implementations will define how to stage blobs, if the default behavior of staging blobs on local disk is not desirable. If a HollowProducer.BlobStager is provided, then neither a blob staging directory or HollowProducer.BlobCompressor should be provided.

An Executor for publishing snapshots

When consumers start up, if the latest announced version does not have a snapshot, they can load an earlier snapshot and follow deltas to get up-to-date. A state can therefore be available and announced prior to the availability of the snapshot. If an Executor is supplied here, then it will be used to publish snapshots. This can be useful if snapshot publishing takes a long time -- subsequent cycles may proceed while snapshot uploads are still in progress.

Number of cycles between snapshots

Because snapshots are not necessary for a data state to be announced, they need not be published every cycle. If this parameter is specified, then a snapshot will be produced only every (n+1)th cycle.

HollowProducer.VersionMinter

Allows for a custom version identifier minting strategy.

Target max type shard size

Specify a target max type shard size. Defaults to 16MB. See http://hollow.how/advanced-topics/#type-sharding

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	HollowProducer.Announcer
static class	HollowProducer.Blob
static interface	HollowProducer.BlobCompressor
static interface	HollowProducer.BlobStager
static class	HollowProducer.BlobStorageCleaner Provides the opportunity to clean the blob storage.
static class	HollowProducer.Builder<B extends HollowProducer.Builder<B>>
static class	HollowProducer.ChecksumValidationException
static class	HollowProducer.Incremental A producer of Hollow data that populates with given data changes (addition, modification or removal), termed incremental production.
static interface	HollowProducer.Populator Represents a task that populates a new data state within a HollowProducer cycle.
static interface	HollowProducer.Publisher
static interface	HollowProducer.ReadState Representation of read state computed from the population of new write state.
static interface	HollowProducer.VersionMinter
static interface	HollowProducer.WriteState Representation of new write state.

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	HollowProducer(HollowProducer.Builder<?> b)
	HollowProducer(HollowProducer.Publisher publisher, HollowProducer.Announcer announcer) Deprecated.

Method Summary

Modifier and Type	Method and Description
void	addListener(HollowProducerEventListener listener) Adds a listener to this producer.
void	addListener(HollowProducerListener listener) Adds a listener to this producer.
boolean	enablePrimaryProducer(boolean doEnable) Invoke this method to alter runCycle behavior.
HollowProducerMetrics	getMetrics()
HollowObjectMapper	getObjectMapper()
HollowWriteStateEngine	getWriteEngine()
void	initializeDataModel(java.lang.Class<?>... classes) Initializes the data model for the given classes.
void	initializeDataModel(HollowSchema... schemas) Initializes the producer data model for the given schemas.
void	removeListener(HollowProducerEventListener listener) Removes a listener to this producer.
void	removeListener(HollowProducerListener listener) Removes a listener to this producer.
HollowProducer.ReadState	restore(long versionDesired, HollowConsumer.BlobRetriever blobRetriever) Restores the data state to a desired version.
long	runCompactionCycle(HollowCompactor.CompactionConfig config) Run a compaction cycle, will produce a data state with exactly the same data as currently, but reorganized so that ordinal holes are filled.
long	runCycle(HollowProducer.Populator task) Runs a cycle to populate, publish, and announce a new single data state.
static HollowProducer.Builder<?>	withPublisher(HollowProducer.Publisher publisher)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HollowProducer

@Deprecated
public HollowProducer(HollowProducer.Publisher publisher,
                                  HollowProducer.Announcer announcer)

Deprecated.

HollowProducer

protected HollowProducer(HollowProducer.Builder<?> b)

Method Detail

getMetrics

public HollowProducerMetrics getMetrics()

Returns:

the metrics for this producer

initializeDataModel

public void initializeDataModel(java.lang.Class<?>... classes)

Initializes the data model for the given classes.

Data model initialization is required prior to restoring the producer. This ensures that restoration can correctly compare the producer's current data model with the data model of the restored data state and manage any differences in those models (such as not restoring state for any types in the restoring data model not present in the producer's current data model).

After initialization a data model initialization event will be emitted to all registered data model initialization listeners listeners.

Parameters:

classes - the data model classes

Throws:

java.lang.IllegalArgumentException - if classes is empty

See Also:

restore(long, HollowConsumer.BlobRetriever)

initializeDataModel

public void initializeDataModel(HollowSchema... schemas)

Initializes the producer data model for the given schemas.

Data model initialization is required prior to restoring the producer. This ensures that restoration can correctly compare the producer's current data model with the data model of the restored data state and manage any differences in those models (such as not restoring state for any types in the restoring data model not present in the producer's current data model).

After initialization a data model initialization event will be emitted to all registered data model initialization listeners listeners.

Parameters:

schemas - the data model classes

Throws:

java.lang.IllegalArgumentException - if schemas is empty

See Also:

restore(long, HollowConsumer.BlobRetriever)

restore

public HollowProducer.ReadState restore(long versionDesired,
                                        HollowConsumer.BlobRetriever blobRetriever)

Restores the data state to a desired version.

Data model initialization is required prior to restoring the producer. This ensures that restoration can correctly compare the producer's current data model with the data model of the restored data state and manage any differences in those models (such as not restoring state for any types in the restoring data model not present in the producer's current data model)

Parameters:

versionDesired - the desired version

blobRetriever - the blob retriever

Returns:

the read state of the restored state

Throws:

java.lang.IllegalStateException - if the producer's data model has not been initialized

See Also:

initializeDataModel(Class[])

getWriteEngine

public HollowWriteStateEngine getWriteEngine()

getObjectMapper

public HollowObjectMapper getObjectMapper()

enablePrimaryProducer

public boolean enablePrimaryProducer(boolean doEnable)

Invoke this method to alter runCycle behavior. If this Producer is not primary, runCycle is a no-op. Note that by default, SingleProducerEnforcer is instantiated as BasicSingleProducerEnforcer, which is initialized to return true for isPrimary()

Parameters:

doEnable - true if enable primary producer, if false

Returns:

true if the intended action was successful

runCycle

public long runCycle(HollowProducer.Populator task)

Runs a cycle to populate, publish, and announce a new single data state.

Parameters:

task - the populating task to add complete state

Returns:

the version identifier of the announced state, otherwise the last successful announced version if 1) there were no data changes compared to that version; or 2) the producer is not the primary producer

Throws:

java.lang.RuntimeException - if the cycle failed

runCompactionCycle

public long runCompactionCycle(HollowCompactor.CompactionConfig config)

Run a compaction cycle, will produce a data state with exactly the same data as currently, but reorganized so that ordinal holes are filled. This may need to be run multiple times to arrive at an optimal state.

Parameters:

config - specifies what criteria to use to determine whether a compaction is necessary

Returns:

the version identifier of the produced state, or AnnouncementWatcher.NO_ANNOUNCEMENT_AVAILABLE if compaction was unnecessary.

addListener

public void addListener(HollowProducerListener listener)

Adds a listener to this producer.

If the listener was previously added to this consumer, as determined by reference equality or Object equality, then this method does nothing.

If a listener is added, concurrently, during the occurrence of a cycle or restore then the listener will not receive events until the next cycle or restore. The listener may also be removed concurrently.

Parameters:

listener - the listener to add

addListener

public void addListener(HollowProducerEventListener listener)

Adds a listener to this producer.

If the listener was previously added to this consumer, as determined by reference equality or Object equality, then this method does nothing.

If a listener is added, concurrently, during the occurrence of a cycle or restore then the listener will not receive events until the next cycle or restore. The listener may also be removed concurrently.

Parameters:

listener - the listener to add

removeListener

public void removeListener(HollowProducerListener listener)

Removes a listener to this producer.

If the listener was not previously added to this producer, as determined by reference equality or Object equality, then this method does nothing.

If a listener is removed, concurrently, during the occurrence of a cycle or restore then the listener will receive all events for that cycle or restore but not receive events for a subsequent cycle or restore.

Parameters:

listener - the listener to remove

removeListener

public void removeListener(HollowProducerEventListener listener)

Removes a listener to this producer.

If the listener was not previously added to this producer, as determined by reference equality or Object equality, then this method does nothing.

If a listener is removed, concurrently, during the occurrence of a cycle or restore then the listener will receive all events for that cycle or restore but not receive events for a subsequent cycle or restore.

Parameters:

listener - the listener to remove

withPublisher

public static HollowProducer.Builder<?> withPublisher(HollowProducer.Publisher publisher)