com.htyleo.extsort

Class ExternalSort

java.lang.Object

com.htyleo.extsort.ExternalSort

public class ExternalSort
extends Object

An implementation of external sorting. The sorting is composed of two phases:

• Partition: partition the input file into a bunch of smaller sorted files.
• Merge: merge the sorted files into a single one, which is the output.

The basic idea of this implementation is that we logically partition the file into smaller slices, each of which is sorted in memory and then written to a separate file. Because sorting slices in memory is performed concurrently (i.e. each slice is sorted in a separate thread), the maximum memory used = max size of thread pool * (slice size + buffer size). This implementation has the following features:

• Every line will be included in some slice in its entirely. In other words, a line will not be sliced in between.
• The first and last several lines of the source file are treated as header and tail. They will be stored in two separate files.
• There is an option to ignore the leading and trailing blank lines in the source file.
• The user-defined line filter and line comparator is supported.

Author:

htyleo

Constructor Summary

Constructors

Constructor and Description
ExternalSort()

Method Summary

Modifier and Type	Method and Description
static void	merge(File dstFile, PartitionResult result, ExternalSortConfig config) External sorting phase One: Merge
static PartitionResult	partition(File sourceFile, File dstDir, ExternalSortConfig config) External sorting phase One: Partition The input is the source file and the output is a bunch of sorted files.
static File	sort(File sourceFile, File dstDir, ExternalSortConfig config) Perform external sorting

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ExternalSort

public ExternalSort()

Method Detail

sort

public static File sort(File sourceFile,
                        File dstDir,
                        ExternalSortConfig config)
                 throws Exception

Perform external sorting

Parameters:

sourceFile - input file

dstDir - output directory

config - sorting configuration

Returns:

output file

Throws:

Exception - If an exception error occurs

partition

public static PartitionResult partition(File sourceFile,
                                        File dstDir,
                                        ExternalSortConfig config)
                                 throws Exception

External sorting phase One: Partition The input is the source file and the output is a bunch of sorted files. Note that the file header and tail will each be stored in a separate file.

Parameters:

sourceFile - input file

dstDir - output directory

config - sorting configuration

Returns:

sorting result

Throws:

Exception - If an exception error occurs

merge

public static void merge(File dstFile,
                         PartitionResult result,
                         ExternalSortConfig config)
                  throws IOException

External sorting phase One: Merge

Parameters:

dstFile - output file

result - partition result

config - sorting configuration

Throws:

IOException - If an I/O error occurs