com.sleepycat.je

Class DiskOrderedCursor

java.lang.Object

com.sleepycat.je.DiskOrderedCursor

All Implemented Interfaces:

ForwardCursor, Closeable, AutoCloseable

public class DiskOrderedCursor
extends Object
implements ForwardCursor

DiskOrderedCursor returns records in unsorted order in exchange for generally faster retrieval times. Instead of key order, an approximation of disk order is used, which results in less I/O. This can be useful when the application needs to scan all records in a database, and will be applying filtering logic which does not need key ordered retrieval. DiskOrderedCursor is created using the Database.openCursor(DiskOrderedCursorConfig) method.

WARNING: After opening a DiskOrderedCursor, deletion of log files by the JE log cleaner will be disabled until close() is called. To prevent unbounded growth of disk usage, be sure to call close() to re-enable log file deletion.

Optional configurations: the following options are available to tune the DiskOrderedCursor.

The DiskOrderedCursor creates a background producer thread which prefetches some target records and inserts them in a queue for use by the cursor. The parameter EnvironmentConfig.DOS_PRODUCER_QUEUE_TIMEOUT applies to this background thread, and controls the timeout which governs the blocking queue.

See DiskOrderedCursorConfig for additional options.

Consistency Guarantees

The consistency guarantees provided by a DiskOrderedCursor are, at best, the same as those provided by READ_UNCOMMITTED (see LockMode). With READ_UNCOMMITTED, changes made by all transactions, including uncommitted transactions, may be returned by the scan. Also, a record returned by the scan is not locked, and may be modified or deleted by the application after it is returned, including modification or deletion of the record at the cursor position.

In the best case, the records returned by the scan correspond to the state of the database (as if READ_UNCOMMITTED were used) at the beginning of the scan plus some, but not all, changes made by the application after the start of the scan. The user should not rely on the scan returning any changes made after the start of the scan. For example, if the record referred to by the DiskOrderedCursor is deleted after the DiskOrderedCursor is positioned at that record, getCurrent() will still return the key and value of that record and OperationStatus.SUCCESS.

In the worst case, rather than corresponding to the state of the database at the beginning of the scan, the records returned by the scan will not include any of the changes made since the last checkpoint. More realistically (in neither the best or worst case), some portion of the changes since the last checkpoint may not be returned by the scan. If it is important to the application that the scan reflects the state of the database at the start of the scan, the following measures should be considered.

• Performing a checkpoint immediately prior to the start of the scan will reduce potential window of changes that may not be returned by scan. The start of the window is the start of checkpoint prior to the scan.
• Increasing the queue size will increase the likelihood that changes made prior to the scan are returned. Of course, a larger queue requires more memory.
• Decreasing the LSN batch size or the internal memory limit will also increase the likelihood that changes made prior to the scan are returned. However, this is also likely to decrease the performance of the scan, since more random IO will probably be required.
• Even by applying the above measures, it is not guaranteed that all changes made prior to the scan are returned. These measures only reduce the frequency that a change made prior to the scan is not returned.

If a transactionally correct data set is required (as defined by READ_COMMITTED), the application must ensure that all transactions that write to the database are committed before the beginning of the scan, and then a checkpoint must be performed before starting the scan. During the checkpoint and the scan, no records in the database of the scan may be inserted, deleted, or modified. While this is possible, it is not the expected use case for a DiskOrderedCursor.

Performance Considerations

The internal algorithm used to approximate disk ordered reads is as follows. An internal producer thread is used to scan the database. This thread is created and started when the DiskOrderedCursor is created, and is destroyed by close(). Scanning consists of two phases. In phase I the in-cache Btree of the scanned database is traversed in key order. The LSNs (physical record addresses) of the data to be fetched are accumulated in a memory buffer. Btree latches are held during the traversal, but only for short durations. In phase II the accumulated LSNs are sorted into disk order, fetched one at a time in that order, and the fetched data is added to a blocking queue. The getNext method in this class removes the next entry from the queue. This approach allows concurrent access to the Database during both phases of the scan, including access by the application's consumer thread (the thread calling getNext).

Phase I does not always process the entire Btree. During phase I if the accumulation of LSNs causes the internal memory limit or LSN batch size to be exceeded, phase I is ended and phase II begins. In this case, after phase II finishes, phase I resumes where it left off in the Btree traversal. Phase I and II are repeated until the entire database is scanned.

By default, the internal memory limit and LSN batch size are unbounded (see DiskOrderedCursorConfig). For a database with a large number of records, this could cause an OutOfMemoryError. Therefore, it is strongly recommended that either the internal memory limit or LSN batch size is configured to limit the use of memory during the scan. On the other hand, the efficiency of the scan is proportional to the amount of memory used. If enough memory is available, the ideal case would be that the database is scanned in in a single iteration of phase I and II. The more iterations, the more random IO will occur.

Another factor is the queue size. During the phase I Btree traversal, data that is resident in the JE cache will be added to the queue immediately, rather than waiting until phase II and fetching it, but only if the queue is not full. Therefore, increasing the size of the queue can avoid fetching data that is resident in the JE cache. Also, increasing the queue size can improve parallelism of the work done by the producer and consumer threads.

Also note that a keys-only scan is much more efficient than the default keys-and-data scan. With a keys-only scan, only the BINs (bottom internal nodes) of the Btree need to be fetched; the LNs (leaf nodes) do not. This is also true of databases configured for duplicates, even for a keys-and-data scan, since internally the key and data are both contained in the BIN.

Since:

5.0

Method Summary

Methods

Modifier and Type	Method and Description
void	close() Discards the cursor.
DiskOrderedCursorConfig	getConfig() Returns this cursor's configuration.
OperationStatus	getCurrent(DatabaseEntry key, DatabaseEntry data, LockMode lockMode) Returns the key/data pair to which the cursor refers.
Database	getDatabase() Returns the Database handle associated with this Cursor.
OperationStatus	getNext(DatabaseEntry key, DatabaseEntry data, LockMode lockMode) Moves the cursor to the next key/data pair and returns that pair.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

getDatabase

public Database getDatabase()

Returns the Database handle associated with this Cursor.

Specified by:

getDatabase in interface ForwardCursor

Returns:

The Database handle associated with this Cursor.

close

public void close()
           throws DatabaseException

Discards the cursor.

The cursor handle may not be used again after this method has been called, regardless of the method's success or failure.

WARNING: To guard against memory leaks, the application should discard all references to the closed handle. While BDB makes an effort to discard references from closed objects to the allocated memory for an environment, this behavior is not guaranteed. The safe course of action for an application is to discard all references to closed BDB objects.

Specified by:

close in interface ForwardCursor

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

EnvironmentFailureException - if an unexpected, internal or environment-wide failure occurs.

DatabaseException

getCurrent

public OperationStatus getCurrent(DatabaseEntry key,
                         DatabaseEntry data,
                         LockMode lockMode)
                           throws DatabaseException

Returns the key/data pair to which the cursor refers.

If this method fails for any reason, the position of the cursor will be unchanged.

Specified by:

getCurrent in interface ForwardCursor

Parameters:

key - the key returned as output. Its byte array does not need to be initialized by the caller.

data - the data returned as output. Its byte array does not need to be initialized by the caller. This argument should be supplied even if the DiskOrderedCursor has been configured with keysOnly.

lockMode - the locking attributes. For DiskOrderedCursors this parameter must be either null or LockMode.READ_UNCOMMITTED since no locking is performed.

Returns:

OperationStatus.KEYEMPTY if there are no more records in the DiskOrderedCursor set, otherwise, OperationStatus.SUCCESS. If the record referred to by a DiskOrderedCursor is deleted after the ForwardCursor is positioned at that record, getCurrent() will still return the key and value of that record and OperationStatus.SUCCESS.

Throws:

OperationFailureException - if one of the Read Operation Failures occurs.

IllegalStateException - if the cursor or database has been closed, or the cursor is uninitialized (not positioned on a record), or the non-transactional cursor was created in a different thread.

IllegalArgumentException - if an invalid parameter is specified.

DatabaseException

getNext

public OperationStatus getNext(DatabaseEntry key,
                      DatabaseEntry data,
                      LockMode lockMode)
                        throws DatabaseException

Moves the cursor to the next key/data pair and returns that pair.

If the cursor is not yet initialized, move the cursor to an arbitrary key/data pair of the database, and return that pair. Otherwise, the cursor is moved to the next key/data pair of the set, and that pair is returned.

If this method fails for any reason, the position of the cursor will be unchanged.

Specified by:

getNext in interface ForwardCursor

Parameters:

key - the key returned as output. Its byte array does not need to be initialized by the caller.

data - the data returned as output. Its byte array does not need to be initialized by the caller. This argument should be supplied even if the DiskOrderedCursor has been configured with keysOnly.

lockMode - the locking attributes. For DiskOrderedCursors this parameter must be either null or LockMode.READ_UNCOMMITTED since no locking is performed.

Returns:

OperationStatus.NOTFOUND if no matching key/data pair is found; otherwise, OperationStatus.SUCCESS.

Throws:

OperationFailureException - if one of the Read Operation Failures occurs.

EnvironmentFailureException - if an unexpected, internal or environment-wide failure occurs.

IllegalStateException - if the cursor or database has been closed, or the non-transactional cursor was created in a different thread.

IllegalArgumentException - if an invalid parameter is specified.

DatabaseException

getConfig

public DiskOrderedCursorConfig getConfig()

Returns this cursor's configuration.

This may differ from the configuration used to open this object if the cursor existed previously.

Returns:

This cursor's configuration.