org.apache.cassandra.db.compaction

Class CompactionManager

java.lang.Object

org.apache.cassandra.db.compaction.CompactionManager

All Implemented Interfaces:

CompactionManagerMBean

public class CompactionManager
extends java.lang.Object
implements CompactionManagerMBean

A singleton which manages a private executor of ongoing compactions.

Scheduling for compaction is accomplished by swapping sstables to be compacted into a set via DataTracker. New scheduling attempts will ignore currently compacting sstables.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CompactionManager.AllSSTableOpStatus
static interface	CompactionManager.CompactionExecutorStatsCollector

Field Summary

Fields

Modifier and Type	Field and Description
static int	GC_ALL
static CompactionManager	instance
static java.lang.ThreadLocal<java.lang.Boolean>	isCompactionManager
static java.lang.String	MBEAN_OBJECT_NAME
static int	NO_GC

Constructor Summary

Constructors

Constructor and Description
CompactionManager()

Method Summary

Modifier and Type	Method and Description
static SSTableWriter	createWriter(ColumnFamilyStore cfs, java.io.File compactionFileLocation, long expectedBloomFilterSize, long repairedAt, SSTableReader sstable)
void	disableAutoCompaction()
void	finishCompactionsAndShutdown(long timeout, java.util.concurrent.TimeUnit unit)
void	forceShutdown() Shutdowns both compaction and validation executors, cancels running compaction / validation, and waits for tasks to complete if tasks were not cancelable.
void	forceUserDefinedCompaction(java.lang.String dataFiles) Triggers the compaction of user specified sstables.
int	getActiveCompactions()
javax.management.openmbean.TabularData	getCompactionHistory() compaction history
java.util.List<java.util.Map<java.lang.String,java.lang.String>>	getCompactions() List of running compaction objects.
java.util.List<java.lang.String>	getCompactionSummary() List of running compaction summary strings.
long	getCompletedTasks()
int	getCoreCompactorThreads() Returns core size of compaction thread pool
int	getCoreValidationThreads() Returns core size of validation thread pool
int	getMaximumCompactorThreads() Returns maximum size of compaction thread pool
int	getMaximumValidatorThreads() Returns size of validator thread pool
int	getPendingTasks()
com.google.common.util.concurrent.RateLimiter	getRateLimiter() Gets compaction rate limiter.
long	getTotalBytesCompacted()
long	getTotalCompactionsCompleted()
void	interruptCompactionFor(java.lang.Iterable<CFMetaData> columnFamilies, boolean interruptValidation) Try to stop all of the compactions for given ColumnFamilies.
void	interruptCompactionForCFs(java.lang.Iterable<ColumnFamilyStore> cfss, boolean interruptValidation)
boolean	isCompacting(java.lang.Iterable<ColumnFamilyStore> cfses)
static boolean	needsCleanup(SSTableReader sstable, java.util.Collection<Range<Token>> ownedRanges) Determines if a cleanup would actually remove any data in this SSTable based on a set of owned ranges.
void	performAnticompaction(ColumnFamilyStore cfs, java.util.Collection<Range<Token>> ranges, Refs<SSTableReader> validatedForRepair, long repairedAt) Make sure the {validatedForRepair} are marked for compaction before calling this.
CompactionManager.AllSSTableOpStatus	performCleanup(ColumnFamilyStore cfStore, int jobs)
void	performMaximal(ColumnFamilyStore cfStore)
CompactionManager.AllSSTableOpStatus	performScrub(ColumnFamilyStore cfs, boolean skipCorrupted, boolean checkData, int jobs)
CompactionManager.AllSSTableOpStatus	performSSTableRewrite(ColumnFamilyStore cfs, boolean excludeCurrentVersion, int jobs)
java.util.List<SSTableReader>	runIndexSummaryRedistribution(IndexSummaryRedistribution redistribution)
void	setCoreCompactorThreads(int number) Allows user to resize maximum size of the compaction thread pool.
void	setCoreValidationThreads(int number) Allows user to resize maximum size of the compaction thread pool.
void	setMaximumCompactorThreads(int number) Allows user to resize maximum size of the compaction thread pool.
void	setMaximumValidatorThreads(int number) Allows user to resize maximum size of the validator thread pool.
void	stopCompaction(java.lang.String type) Stop all running compaction-like tasks having the provided type.
com.google.common.util.concurrent.ListenableFuture<?>	submitAntiCompaction(ColumnFamilyStore cfs, java.util.Collection<Range<Token>> ranges, Refs<SSTableReader> sstables, long repairedAt)
java.util.List<java.util.concurrent.Future<?>>	submitBackground(ColumnFamilyStore cfs) Call this whenever a compaction might be needed on the given columnfamily.
java.util.concurrent.Future<?>	submitCacheWrite(AutoSavingCache.Writer writer)
java.util.concurrent.Future<?>	submitIndexBuild(SecondaryIndexBuilder builder) Is not scheduled, because it is performing disjoint work from sstable compaction.
java.util.List<java.util.concurrent.Future<?>>	submitMaximal(ColumnFamilyStore cfStore, int gcBefore)
java.util.concurrent.Future<?>	submitUserDefined(ColumnFamilyStore cfs, java.util.Collection<Descriptor> dataFiles, int gcBefore)
java.util.concurrent.Future<java.lang.Object>	submitValidation(ColumnFamilyStore cfStore, Validator validator) Does not mutate data, so is not scheduled.
void	waitForCessation(java.lang.Iterable<ColumnFamilyStore> cfss)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MBEAN_OBJECT_NAME

public static final java.lang.String MBEAN_OBJECT_NAME

See Also:

Constant Field Values

instance

public static final CompactionManager instance

NO_GC

public static final int NO_GC

See Also:

Constant Field Values

GC_ALL

public static final int GC_ALL

See Also:

Constant Field Values

isCompactionManager

public static final java.lang.ThreadLocal<java.lang.Boolean> isCompactionManager

Constructor Detail

CompactionManager

public CompactionManager()

Method Detail

getRateLimiter

public com.google.common.util.concurrent.RateLimiter getRateLimiter()

Gets compaction rate limiter. When compaction_throughput_mb_per_sec is 0 or node is bootstrapping, this returns rate limiter with the rate of Double.MAX_VALUE bytes per second. Rate unit is bytes per sec.

Returns:

RateLimiter with rate limit set

submitBackground

public java.util.List<java.util.concurrent.Future<?>> submitBackground(ColumnFamilyStore cfs)

Call this whenever a compaction might be needed on the given columnfamily. It's okay to over-call (within reason) if a call is unnecessary, it will turn into a no-op in the bucketing/candidate-scan phase.

isCompacting

public boolean isCompacting(java.lang.Iterable<ColumnFamilyStore> cfses)

forceShutdown

public void forceShutdown()

Shutdowns both compaction and validation executors, cancels running compaction / validation, and waits for tasks to complete if tasks were not cancelable.

finishCompactionsAndShutdown

public void finishCompactionsAndShutdown(long timeout,
                                         java.util.concurrent.TimeUnit unit)
                                  throws java.lang.InterruptedException

Throws:

java.lang.InterruptedException

performScrub

public CompactionManager.AllSSTableOpStatus performScrub(ColumnFamilyStore cfs,
                                                         boolean skipCorrupted,
                                                         boolean checkData,
                                                         int jobs)
                                                  throws java.lang.InterruptedException,
                                                         java.util.concurrent.ExecutionException

Throws:

java.lang.InterruptedException

java.util.concurrent.ExecutionException

performSSTableRewrite

public CompactionManager.AllSSTableOpStatus performSSTableRewrite(ColumnFamilyStore cfs,
                                                                  boolean excludeCurrentVersion,
                                                                  int jobs)
                                                           throws java.lang.InterruptedException,
                                                                  java.util.concurrent.ExecutionException

Throws:

java.lang.InterruptedException

java.util.concurrent.ExecutionException

performCleanup

public CompactionManager.AllSSTableOpStatus performCleanup(ColumnFamilyStore cfStore,
                                                           int jobs)
                                                    throws java.lang.InterruptedException,
                                                           java.util.concurrent.ExecutionException

Throws:

java.lang.InterruptedException

java.util.concurrent.ExecutionException

submitAntiCompaction

public com.google.common.util.concurrent.ListenableFuture<?> submitAntiCompaction(ColumnFamilyStore cfs,
                                                                                  java.util.Collection<Range<Token>> ranges,
                                                                                  Refs<SSTableReader> sstables,
                                                                                  long repairedAt)

performAnticompaction

public void performAnticompaction(ColumnFamilyStore cfs,
                                  java.util.Collection<Range<Token>> ranges,
                                  Refs<SSTableReader> validatedForRepair,
                                  long repairedAt)
                           throws java.lang.InterruptedException,
                                  java.util.concurrent.ExecutionException,
                                  java.io.IOException

Make sure the {validatedForRepair} are marked for compaction before calling this. Caller must reference the validatedForRepair sstables (via ParentRepairSession.getActiveRepairedSSTableRefsForAntiCompaction(..)).

Parameters:

cfs -

ranges - Ranges that the repair was carried out on

validatedForRepair - SSTables containing the repaired ranges. Should be referenced before passing them.

Throws:

InterruptedException, - ExecutionException, IOException

java.lang.InterruptedException

java.util.concurrent.ExecutionException

java.io.IOException

performMaximal

public void performMaximal(ColumnFamilyStore cfStore)
                    throws java.lang.InterruptedException,
                           java.util.concurrent.ExecutionException

Throws:

java.lang.InterruptedException

java.util.concurrent.ExecutionException

submitMaximal

public java.util.List<java.util.concurrent.Future<?>> submitMaximal(ColumnFamilyStore cfStore,
                                                                    int gcBefore)

forceUserDefinedCompaction

public void forceUserDefinedCompaction(java.lang.String dataFiles)

Description copied from interface: CompactionManagerMBean

Triggers the compaction of user specified sstables. You can specify files from various keyspaces and columnfamilies. If you do so, user defined compaction is performed several times to the groups of files in the same keyspace/columnfamily.

Specified by:

forceUserDefinedCompaction in interface CompactionManagerMBean

Parameters:

dataFiles - a comma separated list of sstable filename to compact

submitUserDefined

public java.util.concurrent.Future<?> submitUserDefined(ColumnFamilyStore cfs,
                                                        java.util.Collection<Descriptor> dataFiles,
                                                        int gcBefore)

submitValidation

public java.util.concurrent.Future<java.lang.Object> submitValidation(ColumnFamilyStore cfStore,
                                                                      Validator validator)

Does not mutate data, so is not scheduled.

disableAutoCompaction

public void disableAutoCompaction()

needsCleanup

public static boolean needsCleanup(SSTableReader sstable,
                                   java.util.Collection<Range<Token>> ownedRanges)

Determines if a cleanup would actually remove any data in this SSTable based on a set of owned ranges.

createWriter

public static SSTableWriter createWriter(ColumnFamilyStore cfs,
                                         java.io.File compactionFileLocation,
                                         long expectedBloomFilterSize,
                                         long repairedAt,
                                         SSTableReader sstable)

submitIndexBuild

public java.util.concurrent.Future<?> submitIndexBuild(SecondaryIndexBuilder builder)

Is not scheduled, because it is performing disjoint work from sstable compaction.

submitCacheWrite

public java.util.concurrent.Future<?> submitCacheWrite(AutoSavingCache.Writer writer)

runIndexSummaryRedistribution

public java.util.List<SSTableReader> runIndexSummaryRedistribution(IndexSummaryRedistribution redistribution)
                                                            throws java.io.IOException

Throws:

java.io.IOException

getActiveCompactions

public int getActiveCompactions()

getCompactions

public java.util.List<java.util.Map<java.lang.String,java.lang.String>> getCompactions()

Description copied from interface: CompactionManagerMBean

List of running compaction objects.

Specified by:

getCompactions in interface CompactionManagerMBean

getCompactionSummary

public java.util.List<java.lang.String> getCompactionSummary()

Description copied from interface: CompactionManagerMBean

List of running compaction summary strings.

Specified by:

getCompactionSummary in interface CompactionManagerMBean

getCompactionHistory

public javax.management.openmbean.TabularData getCompactionHistory()

Description copied from interface: CompactionManagerMBean

compaction history

Specified by:

getCompactionHistory in interface CompactionManagerMBean

getTotalBytesCompacted

public long getTotalBytesCompacted()

Specified by:

getTotalBytesCompacted in interface CompactionManagerMBean

Returns:

total number of bytes compacted since server [re]start

See Also:

CompactionMetrics.bytesCompacted

getTotalCompactionsCompleted

public long getTotalCompactionsCompleted()

Specified by:

getTotalCompactionsCompleted in interface CompactionManagerMBean

Returns:

total number of compactions since server [re]start

See Also:

CompactionMetrics.totalCompactionsCompleted

getPendingTasks

public int getPendingTasks()

Specified by:

getPendingTasks in interface CompactionManagerMBean

Returns:

estimated number of compactions remaining to perform

See Also:

CompactionMetrics.pendingTasks

getCompletedTasks

public long getCompletedTasks()

Specified by:

getCompletedTasks in interface CompactionManagerMBean

Returns:

number of completed compactions since server [re]start

See Also:

CompactionMetrics.completedTasks

stopCompaction

public void stopCompaction(java.lang.String type)

Description copied from interface: CompactionManagerMBean

Stop all running compaction-like tasks having the provided type.

Specified by:

stopCompaction in interface CompactionManagerMBean

Parameters:

type - the type of compaction to stop. Can be one of: - COMPACTION - VALIDATION - CLEANUP - SCRUB - INDEX_BUILD

getCoreCompactorThreads

public int getCoreCompactorThreads()

Description copied from interface: CompactionManagerMBean

Returns core size of compaction thread pool

Specified by:

getCoreCompactorThreads in interface CompactionManagerMBean

setCoreCompactorThreads

public void setCoreCompactorThreads(int number)

Description copied from interface: CompactionManagerMBean

Allows user to resize maximum size of the compaction thread pool.

Specified by:

setCoreCompactorThreads in interface CompactionManagerMBean

Parameters:

number - New maximum of compaction threads

getMaximumCompactorThreads

public int getMaximumCompactorThreads()

Description copied from interface: CompactionManagerMBean

Returns maximum size of compaction thread pool

Specified by:

getMaximumCompactorThreads in interface CompactionManagerMBean

setMaximumCompactorThreads

public void setMaximumCompactorThreads(int number)

Description copied from interface: CompactionManagerMBean

Allows user to resize maximum size of the compaction thread pool.

Specified by:

setMaximumCompactorThreads in interface CompactionManagerMBean

Parameters:

number - New maximum of compaction threads

getCoreValidationThreads

public int getCoreValidationThreads()

Description copied from interface: CompactionManagerMBean

Returns core size of validation thread pool

Specified by:

getCoreValidationThreads in interface CompactionManagerMBean

setCoreValidationThreads

public void setCoreValidationThreads(int number)

Description copied from interface: CompactionManagerMBean

Allows user to resize maximum size of the compaction thread pool.

Specified by:

setCoreValidationThreads in interface CompactionManagerMBean

Parameters:

number - New maximum of compaction threads

getMaximumValidatorThreads

public int getMaximumValidatorThreads()

Description copied from interface: CompactionManagerMBean

Returns size of validator thread pool

Specified by:

getMaximumValidatorThreads in interface CompactionManagerMBean

setMaximumValidatorThreads

public void setMaximumValidatorThreads(int number)

Description copied from interface: CompactionManagerMBean

Allows user to resize maximum size of the validator thread pool.

Specified by:

setMaximumValidatorThreads in interface CompactionManagerMBean

Parameters:

number - New maximum of validator threads

interruptCompactionFor

public void interruptCompactionFor(java.lang.Iterable<CFMetaData> columnFamilies,
                                   boolean interruptValidation)

Try to stop all of the compactions for given ColumnFamilies. Note that this method does not wait for all compactions to finish; you'll need to loop against isCompacting if you want that behavior.

Parameters:

columnFamilies - The ColumnFamilies to try to stop compaction upon.

interruptValidation - true if validation operations for repair should also be interrupted

interruptCompactionForCFs

public void interruptCompactionForCFs(java.lang.Iterable<ColumnFamilyStore> cfss,
                                      boolean interruptValidation)

waitForCessation

public void waitForCessation(java.lang.Iterable<ColumnFamilyStore> cfss)