org.apache.drill.exec.physical.impl.PartitionLimit

Class TestPartitionLimitBatch

java.lang.Object

DrillTest

org.apache.drill.exec.ExecTest

org.apache.drill.test.PhysicalOpUnitTestBase

org.apache.drill.exec.physical.impl.BaseTestOpBatchEmitOutcome

org.apache.drill.exec.physical.impl.PartitionLimit.TestPartitionLimitBatch

public class TestPartitionLimitBatch
extends BaseTestOpBatchEmitOutcome

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.drill.test.PhysicalOpUnitTestBase

PhysicalOpUnitTestBase.BatchIterator, PhysicalOpUnitTestBase.MockExecutorFragmentContext, PhysicalOpUnitTestBase.MockPhysicalOperator

Field Summary

Fields inherited from class org.apache.drill.exec.physical.impl.BaseTestOpBatchEmitOutcome

emptyInputRowSet, inputContainer, inputContainerSv2, inputOutcomes, inputSchema, nonEmptyInputRowSet, outputRecordCount

Fields inherited from class org.apache.drill.test.PhysicalOpUnitTestBase

classpathScan, dirTestWatcher, drillbitContext, drillConf, fragContext, opContext, opCreatorReg, operatorFixture, scanDecodeExecutor, scanExecutor

Fields inherited from class org.apache.drill.exec.ExecTest

c, optionManager

Constructor Summary

Constructors

Constructor and Description
TestPartitionLimitBatch()

Method Summary

Modifier and Type	Method and Description
void	afterTestCleanup() Cleanup method executed post each test
static void	partitionLimitSetup()
void	testPartitionLimit_AcrossEmitOutcome() Verifies that PartitionLimitRecordBatch refreshes it's state after seeing first EMIT outcome and works on data batches following it as new set's of incoming batch and apply the partition limit rule from fresh on those.
void	testPartitionLimit_EmptyBatchEmitOutcome() Verifies that empty batch with both OK_NEW_SCHEMA and EMIT outcome is not ignored by PartitionLimitRecordBatch and is passed to the downstream operator.
void	testPartitionLimit_IgnoreOnePartitionIdWithOffset() Verifies PartitionLimitRecordBatch works correctly in cases where start offset is such that all the records of a partition id is ignored but records in other partition id is selected.
void	testPartitionLimit_LargeOffsetIgnoreAllRecords()
void	testPartitionLimit_Limit0() Verifies PartitionLimitRecordBatch works correctly when start and end offset is same.
void	testPartitionLimit_MultipleEmit_SingleMultipleBatch_WithSV2_FilteredRows() Verifies PartitionLimitRecordBatch behaves correctly across EMIT boundary with single or multiple batches (with sv2) within each EMIT boundary.
void	testPartitionLimit_MultipleEmit_SingleMultipleBatch_WithSV2() Verifies PartitionLimitRecordBatch behaves correctly across EMIT boundary with single or multiple batches (with sv2) within each EMIT boundary.
void	testPartitionLimit_MultipleEmit_SingleMultipleBatch() Verifies PartitionLimitRecordBatch behaves correctly across EMIT boundary with single or multiple batches within each EMIT boundary.
void	testPartitionLimit_NegativeOffset() Verifies PartitionLimitRecordBatch takes care of provided negative start offset correctly
void	testPartitionLimit_NoLimit() Verifies PartitionLimitRecordBatch works correctly for cases where no end offset is mentioned.
void	testPartitionLimit_NonEmptyBatchEmitOutcome() Verifies PartitionLimitRecordBatch considers all the batch until it sees EMIT outcome and return output batch with data that meets the PartitionLimitRecordBatch criteria.
void	testPartitionLimit_NonEmptyFirst_EmptyOKEmitOutcome() Verifies that when the PartitionLimitRecordBatch number of records is found with first incoming batch, then next empty incoming batch with OK outcome is ignored, but the empty EMIT outcome batch is not ignored.
void	testPartitionLimit_PartitionIdSelectedAcrossBatches() Verifies PartitionLimitRecordBatch works correctly in cases a partition id spans across batches and limit condition is met by picking records from multiple batch for same partition id.
void	testPartitionLimit_PartitionIdSpanningAcrossBatches_WithOffset()
void	testPartitionLimit_PartitionIdSpanningAcrossBatches() Verifies that PartitionLimitRecordBatch considers same partition id across batches but within EMIT boundary to impose limit condition.
void	testPartitionLimit_ResetsAfterFirstEmitOutcome() Verifies that PartitionLimitRecordBatch batch operates on batches across EMIT boundary with fresh configuration.

Methods inherited from class org.apache.drill.exec.physical.impl.BaseTestOpBatchEmitOutcome

afterTest, beforeTest, setUpBeforeClass

Methods inherited from class org.apache.drill.test.PhysicalOpUnitTestBase

getJsonReadersFromBatchString, getJsonReadersFromInputFiles, getOpCreatorReg, getReaderListForJsonBatches, joinCond, legacyOpTestBuilder, mockFragmentContext, mockOpContext, opTestBuilder, ordering, parseExprs, setup, teardown

Methods inherited from class org.apache.drill.exec.ExecTest

clear, getLocalFileSystem, mockDrillbitContext, mockUsDateFormatSymbols, mockUtcDateTimeZone, parseExpr, setupOptionManager

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TestPartitionLimitBatch

public TestPartitionLimitBatch()

Method Detail

partitionLimitSetup

public static void partitionLimitSetup()

afterTestCleanup

public void afterTestCleanup()

Cleanup method executed post each test

testPartitionLimit_EmptyBatchEmitOutcome

public void testPartitionLimit_EmptyBatchEmitOutcome()

Verifies that empty batch with both OK_NEW_SCHEMA and EMIT outcome is not ignored by PartitionLimitRecordBatch and is passed to the downstream operator.

testPartitionLimit_NonEmptyBatchEmitOutcome

public void testPartitionLimit_NonEmptyBatchEmitOutcome()

Verifies PartitionLimitRecordBatch considers all the batch until it sees EMIT outcome and return output batch with data that meets the PartitionLimitRecordBatch criteria.

testPartitionLimit_ResetsAfterFirstEmitOutcome

public void testPartitionLimit_ResetsAfterFirstEmitOutcome()

Verifies that PartitionLimitRecordBatch batch operates on batches across EMIT boundary with fresh configuration. That is it considers partition column data separately for batches across EMIT boundary.

testPartitionLimit_NonEmptyFirst_EmptyOKEmitOutcome

public void testPartitionLimit_NonEmptyFirst_EmptyOKEmitOutcome()

Verifies that when the PartitionLimitRecordBatch number of records is found with first incoming batch, then next empty incoming batch with OK outcome is ignored, but the empty EMIT outcome batch is not ignored. Empty incoming batch with EMIT outcome produces empty output batch with EMIT outcome.

testPartitionLimit_AcrossEmitOutcome

public void testPartitionLimit_AcrossEmitOutcome()

Verifies that PartitionLimitRecordBatch refreshes it's state after seeing first EMIT outcome and works on data batches following it as new set's of incoming batch and apply the partition limit rule from fresh on those. So for first set of batches with OK_NEW_SCHEMA and EMIT outcome the total number of records received being less than limit condition, it still produces an output with that many records for each partition key (in this case 1 even though limit number of records is 2). After seeing EMIT, it refreshes it's state and operate on next input batches to again return limit number of records per partition id. So for 3rd batch with 6 records and 3 partition id and with EMIT outcome it produces an output batch with <=2 records for each partition id.

testPartitionLimit_PartitionIdSpanningAcrossBatches

public void testPartitionLimit_PartitionIdSpanningAcrossBatches()

Verifies that PartitionLimitRecordBatch considers same partition id across batches but within EMIT boundary to impose limit condition.

testPartitionLimit_PartitionIdSpanningAcrossBatches_WithOffset

public void testPartitionLimit_PartitionIdSpanningAcrossBatches_WithOffset()

testPartitionLimit_PartitionIdSelectedAcrossBatches

public void testPartitionLimit_PartitionIdSelectedAcrossBatches()

Verifies PartitionLimitRecordBatch works correctly in cases a partition id spans across batches and limit condition is met by picking records from multiple batch for same partition id.

testPartitionLimit_IgnoreOnePartitionIdWithOffset

public void testPartitionLimit_IgnoreOnePartitionIdWithOffset()

Verifies PartitionLimitRecordBatch works correctly in cases where start offset is such that all the records of a partition id is ignored but records in other partition id is selected.

testPartitionLimit_LargeOffsetIgnoreAllRecords

public void testPartitionLimit_LargeOffsetIgnoreAllRecords()

testPartitionLimit_Limit0

public void testPartitionLimit_Limit0()

Verifies PartitionLimitRecordBatch works correctly when start and end offset is same. In this case it works as Limit 0 scenario where it will not output any rows for any partition id across batches.

testPartitionLimit_NoLimit

public void testPartitionLimit_NoLimit()

Verifies PartitionLimitRecordBatch works correctly for cases where no end offset is mentioned. This necessary means selecting all the records in a partition.

testPartitionLimit_NegativeOffset

public void testPartitionLimit_NegativeOffset()

Verifies PartitionLimitRecordBatch takes care of provided negative start offset correctly

testPartitionLimit_MultipleEmit_SingleMultipleBatch

public void testPartitionLimit_MultipleEmit_SingleMultipleBatch()

Verifies PartitionLimitRecordBatch behaves correctly across EMIT boundary with single or multiple batches within each EMIT boundary. It resets it states correctly across EMIT boundary and then operates on all the batches within EMIT boundary at a time.

testPartitionLimit_MultipleEmit_SingleMultipleBatch_WithSV2

public void testPartitionLimit_MultipleEmit_SingleMultipleBatch_WithSV2()

Verifies PartitionLimitRecordBatch behaves correctly across EMIT boundary with single or multiple batches (with sv2) within each EMIT boundary. It resets it states correctly across EMIT boundary and then operates on all the batches within EMIT boundary at a time.

testPartitionLimit_MultipleEmit_SingleMultipleBatch_WithSV2_FilteredRows

public void testPartitionLimit_MultipleEmit_SingleMultipleBatch_WithSV2_FilteredRows()

Verifies PartitionLimitRecordBatch behaves correctly across EMIT boundary with single or multiple batches (with sv2) within each EMIT boundary. It resets it states correctly across EMIT boundary and then operates on all the batches within EMIT boundary at a time.