Class JdbcQueue<ID,DATA>
- java.lang.Object
-
- com.github.ddth.queue.impl.AbstractQueue<ID,DATA>
-
- com.github.ddth.queue.impl.AbstractEphemeralSupportQueue<ID,DATA>
-
- com.github.ddth.queue.impl.JdbcQueue<ID,DATA>
-
- All Implemented Interfaces:
IQueue<ID,DATA>,AutoCloseable
- Direct Known Subclasses:
BaseUniversalJdbcQueue
public abstract class JdbcQueue<ID,DATA> extends AbstractEphemeralSupportQueue<ID,DATA>
Abstract JDBC implementation ofIQueue.Features:
- Queue-size support: yes
- Ephemeral storage support: yes
- Ephemeral-size support: yes
Implementation:
- Queue storage & Ephemeral storage are 2 database tables, same structure!
take():- Obtain a database
Connection. - Check if
ephemeral sizeis full or not. - Call
_takeWithRetries(Connection, int, int); which, in turn:- Start a database transaction.
- Read a message off queue storage by calling
peekFromQueueStorage(Connection). - Remove the message from queue (call
removeFromQueueStorage(Connection, IQueueMessage)) and out to ephemeral storage (callputToEphemeralStorage(Connection, IQueueMessage)). - Commit transaction if all above operations were successful, rollback transaction otherwise.
- Obtain a database
finish(IQueueMessage):- Obtain a database
Connection. - Call
_finishWithRetries(Connection, IQueueMessage, int, int); which, in turn:
- Obtain a database
getOrphanMessages(long):- Obtain a database
Connection. - Call
_getOrphanMessagesWithRetries(long, Connection, int, int); which, in turn:- Start a database transaction.
- Call
getOrphanMessagesFromEphemeralStorage(Connection, long). - Commit transaction.
- Obtain a database
AbstractQueue.queue(IQueueMessage),AbstractQueue.requeue(IQueueMessage)andAbstractQueue.requeueSilent(IQueueMessage): inherit fromAbstractQueueand overridedoPutToQueue(IQueueMessage, PutToQueueCase)- If
queueCaseparameter isAbstractQueue.PutToQueueCase.REQUEUE, call_requeueWithRetries(Connection, IQueueMessage, int, int); which, in turn:- Start a database transactions.
- Call
removeFromEphemeralStorage(Connection, IQueueMessage). - Call
putToQueueStorage(Connection, IQueueMessage). - Commit transaction.
- If
queueCaseparameter isAbstractQueue.PutToQueueCase.REQUEUE_SILENT, call_requeueSilentWithRetries(Connection, IQueueMessage, int, int); which, in turn:- Start a database transactions.
- Call
removeFromEphemeralStorage(Connection, IQueueMessage). - Call
putToQueueStorage(Connection, IQueueMessage). - Commit transaction.
- Otherwise, call
_queueWithRetries(Connection, IQueueMessage, int, int); which, in turn:
- If
- Since:
- 0.1.0
- Author:
- Thanh Ba Nguyen
-
-
Nested Class Summary
-
Nested classes/interfaces inherited from class com.github.ddth.queue.impl.AbstractQueue
AbstractQueue.PutToQueueCase
-
-
Field Summary
Fields Modifier and Type Field Description static intDEFAULT_MAX_RETRIESstatic intDEFAULT_TRANX_ISOLATION_LEVEL-
Fields inherited from interface com.github.ddth.queue.IQueue
SIZE_NOT_SUPPORTED
-
-
Constructor Summary
Constructors Constructor Description JdbcQueue()
-
Method Summary
All Methods Instance Methods Abstract Methods Concrete Methods Deprecated Methods Modifier and Type Method Description protected void_finishWithRetries(Connection conn, IQueueMessage<ID,DATA> msg, int numRetries, int maxRetries)Perform "finish" action, retry if deadlock.protected Collection<? extends IQueueMessage<ID,DATA>>_getOrphanMessagesWithRetries(long thresholdTimestampMs, Connection conn, int numRetries, int maxRetries)Get all orphan messages (messages that were left in ephemeral storage for a long time), retry if deadlock.protected boolean_queueWithRetries(Connection conn, IQueueMessage<ID,DATA> immutableMsg, int numRetries, int maxRetries)Queue a message, retry if deadlock.protected boolean_requeueSilentWithRetries(Connection conn, IQueueMessage<ID,DATA> immutableMsg, int numRetries, int maxRetries)Re-queue (silentlt) a message, retry if deadlock.protected boolean_requeueWithRetries(Connection conn, IQueueMessage<ID,DATA> immutableMsg, int numRetries, int maxRetries)Re-queue a message, retry if deadlock.protected IQueueMessage<ID,DATA>_takeWithRetries(Connection conn, int numRetries, int maxRetries)Take a message from queue, retry if deadlock.protected com.github.ddth.dao.jdbc.IJdbcHelperbuildJdbcHelper()Build anIJdbcHelperto be used by this JDBC queue.voiddestroy()Destroy method.protected booleandoPutToQueue(IQueueMessage<ID,DATA> msg, AbstractQueue.PutToQueueCase queueCase)Put a message to the queue storage.protected intdoSize(Connection conn, String sql, String field, Object... params)intephemeralSize()Get number of items currently in ephemeral storage.protected intephemeralSize(Connection conn)Get number of items currently in ephemeral storage.protected intexecuteWithRetries(int numRetries, int maxRetries, boolean transactionMode, Connection conn, String sql, Object... params)Execute a query, retry if deadlock.protected intexecuteWithRetries(int numRetries, int maxRetries, boolean transactionMode, Connection conn, String sql, Map<String,Object> params)Execute a query, retry if deadlock.protected <T> TexecuteWithRetries(int numRetries, int maxRetries, Supplier<T> sqlRunner)Execute queries, retry if deadlock.voidfinish(IQueueMessage<ID,DATA> msg)Called when finish processing the message to cleanup ephemeral storage.DataSourcegetDataSource()Getter fordataSource.com.github.ddth.dao.jdbc.IJdbcHelpergetJdbcHelper()Getter forjdbcHelper.intgetMaxRetries()Max number of retires for DB-operations.Collection<IQueueMessage<ID,DATA>>getOrphanMessages(long thresholdTimestampMs)Get all orphan messages (messages that were left in ephemeral storage for a long time).protected abstract Collection<? extends IQueueMessage<ID,DATA>>getOrphanMessagesFromEphemeralStorage(Connection conn, long thresholdTimestampMs)Primitive operation: Get all orphan messages (messages that were left in ephemeral storage for a long time).StringgetTableName()Name of database table to store queue messages.StringgetTableNameEphemeral()Name of database table to store ephemeral messages.intgetTransactionIsolationLevel()Transaction isolation level used in DB-operations.JdbcQueue<ID,DATA>init()Initializing method.protected abstract IQueueMessage<ID,DATA>peekFromQueueStorage(Connection conn)Primitive operation: Read (but not removed) message from head of queue storage.protected abstract booleanputToEphemeralStorage(Connection conn, IQueueMessage<ID,DATA> msg)Primitive operation: Put a message to the ephemeral storage.protected abstract booleanputToQueueStorage(Connection conn, IQueueMessage<ID,DATA> msg)Primitive operation: Put a message to tail of the queue storage.intqueueSize()Get number of items currently in queue storage.protected intqueueSize(Connection conn)Get number of items currently in queue storage.protected IQueueMessage<ID,DATA>readFromEphemeralStorage(Connection conn, IQueueMessage<ID,DATA> msg)Deprecated.deprecated since 1.0.0, usereadFromEphemeralStorage(Connection, Object)protected abstract IQueueMessage<ID,DATA>readFromEphemeralStorage(Connection conn, ID id)Primitive operation: Read (but not removed) a message from the ephemeral storage.protected abstract booleanremoveFromEphemeralStorage(Connection conn, IQueueMessage<ID,DATA> msg)Primitive operation: Remove a message from the queue storage.protected abstract booleanremoveFromQueueStorage(Connection conn, IQueueMessage<ID,DATA> msg)Primitive operation: Remove a message from the queue storage.JdbcQueue<ID,DATA>setDataSource(DataSource dataSource)Setter fordataSource.JdbcQueue<ID,DATA>setJdbcHelper(com.github.ddth.dao.jdbc.IJdbcHelper jdbcHelper)Setter forjdbcHelper.protected JdbcQueue<ID,DATA>setJdbcHelper(com.github.ddth.dao.jdbc.IJdbcHelper jdbcHelper, boolean setMyOwnJdbcHelper)Setter forjdbcHelper.JdbcQueue<ID,DATA>setMaxRetries(int maxRetries)Max number of retires for DB-operations.JdbcQueue<ID,DATA>setTableName(String tableName)Name of database table to store queue messages.JdbcQueue<ID,DATA>setTableNameEphemeral(String tableNameEphemeral)Name of database table to store ephemeral messages.JdbcQueue<ID,DATA>setTransactionIsolationLevel(int transactionIsolationLevel)Transaction isolation level used in DB-operations.IQueueMessage<ID,DATA>take()Take a message out of queue.-
Methods inherited from class com.github.ddth.queue.impl.AbstractEphemeralSupportQueue
getEphemeralDisabled, getEphemeralMaxSize, isEphemeralDisabled, setEphemeralDisabled, setEphemeralMaxSize
-
Methods inherited from class com.github.ddth.queue.impl.AbstractQueue
close, createMessage, createMessage, createMessage, deserialize, deserialize, getMessageFactory, getQueueName, getSerDeser, queue, requeue, requeueSilent, serialize, setMessageFactory, setQueueName, setSerDeser
-
-
-
-
Field Detail
-
DEFAULT_MAX_RETRIES
public static final int DEFAULT_MAX_RETRIES
- See Also:
- Constant Field Values
-
DEFAULT_TRANX_ISOLATION_LEVEL
public static final int DEFAULT_TRANX_ISOLATION_LEVEL
- See Also:
- Constant Field Values
-
-
Method Detail
-
setTableName
public JdbcQueue<ID,DATA> setTableName(String tableName)
Name of database table to store queue messages.- Parameters:
tableName-- Returns:
-
getTableName
public String getTableName()
Name of database table to store queue messages.- Returns:
-
setTableNameEphemeral
public JdbcQueue<ID,DATA> setTableNameEphemeral(String tableNameEphemeral)
Name of database table to store ephemeral messages.- Parameters:
tableNameEphemeral-- Returns:
-
getTableNameEphemeral
public String getTableNameEphemeral()
Name of database table to store ephemeral messages.- Returns:
-
setTransactionIsolationLevel
public JdbcQueue<ID,DATA> setTransactionIsolationLevel(int transactionIsolationLevel)
Transaction isolation level used in DB-operations.- Parameters:
transactionIsolationLevel-- Returns:
-
getTransactionIsolationLevel
public int getTransactionIsolationLevel()
Transaction isolation level used in DB-operations.- Returns:
-
getJdbcHelper
public com.github.ddth.dao.jdbc.IJdbcHelper getJdbcHelper()
Getter forjdbcHelper.- Returns:
- Since:
- 0.5.0
-
setJdbcHelper
protected JdbcQueue<ID,DATA> setJdbcHelper(com.github.ddth.dao.jdbc.IJdbcHelper jdbcHelper, boolean setMyOwnJdbcHelper)
Setter forjdbcHelper.- Parameters:
jdbcHelper-setMyOwnJdbcHelper-- Returns:
- Since:
- 0.7.1
-
setJdbcHelper
public JdbcQueue<ID,DATA> setJdbcHelper(com.github.ddth.dao.jdbc.IJdbcHelper jdbcHelper)
Setter forjdbcHelper.- Parameters:
jdbcHelper-- Returns:
- Since:
- 0.5.1.1
-
getDataSource
public DataSource getDataSource()
Getter fordataSource.- Returns:
- Since:
- 0.6.2.5
-
setDataSource
public JdbcQueue<ID,DATA> setDataSource(DataSource dataSource)
Setter fordataSource.- Parameters:
dataSource-- Returns:
- Since:
- 0.6.2.5
-
setMaxRetries
public JdbcQueue<ID,DATA> setMaxRetries(int maxRetries)
Max number of retires for DB-operations.- Parameters:
maxRetries-- Returns:
-
getMaxRetries
public int getMaxRetries()
Max number of retires for DB-operations.- Returns:
-
buildJdbcHelper
protected com.github.ddth.dao.jdbc.IJdbcHelper buildJdbcHelper()
Build anIJdbcHelperto be used by this JDBC queue.- Returns:
- Since:
- 0.6.2.6
-
init
public JdbcQueue<ID,DATA> init() throws Exception
Initializing method.- Overrides:
initin classAbstractQueue<ID,DATA>- Returns:
- Throws:
Exception
-
destroy
public void destroy()
Destroy method.- Overrides:
destroyin classAbstractQueue<ID,DATA>
-
peekFromQueueStorage
protected abstract IQueueMessage<ID,DATA> peekFromQueueStorage(Connection conn)
Primitive operation: Read (but not removed) message from head of queue storage.- Parameters:
conn-- Returns:
-
readFromEphemeralStorage
protected IQueueMessage<ID,DATA> readFromEphemeralStorage(Connection conn, IQueueMessage<ID,DATA> msg)
Deprecated.deprecated since 1.0.0, usereadFromEphemeralStorage(Connection, Object)Primitive operation: Read (but not removed) a message from the ephemeral storage.- Parameters:
conn-msg-- Returns:
- Since:
- 0.2.1
-
readFromEphemeralStorage
protected abstract IQueueMessage<ID,DATA> readFromEphemeralStorage(Connection conn, ID id)
Primitive operation: Read (but not removed) a message from the ephemeral storage.- Parameters:
conn-id-- Returns:
- Since:
- 1.0.0
-
getOrphanMessagesFromEphemeralStorage
protected abstract Collection<? extends IQueueMessage<ID,DATA>> getOrphanMessagesFromEphemeralStorage(Connection conn, long thresholdTimestampMs)
Primitive operation: Get all orphan messages (messages that were left in ephemeral storage for a long time).- Parameters:
conn-thresholdTimestampMs- get all orphan messages that were queued before this timestamp- Returns:
- Since:
- 0.2.0
-
putToQueueStorage
protected abstract boolean putToQueueStorage(Connection conn, IQueueMessage<ID,DATA> msg)
Primitive operation: Put a message to tail of the queue storage.- Parameters:
conn-msg-- Returns:
-
putToEphemeralStorage
protected abstract boolean putToEphemeralStorage(Connection conn, IQueueMessage<ID,DATA> msg)
Primitive operation: Put a message to the ephemeral storage.- Parameters:
conn-msg-
-
removeFromQueueStorage
protected abstract boolean removeFromQueueStorage(Connection conn, IQueueMessage<ID,DATA> msg)
Primitive operation: Remove a message from the queue storage.- Parameters:
conn-msg-- Returns:
-
removeFromEphemeralStorage
protected abstract boolean removeFromEphemeralStorage(Connection conn, IQueueMessage<ID,DATA> msg)
Primitive operation: Remove a message from the queue storage.- Parameters:
conn-msg-- Returns:
-
executeWithRetries
protected int executeWithRetries(int numRetries, int maxRetries, boolean transactionMode, Connection conn, String sql, Map<String,Object> params)Execute a query, retry if deadlock.Note: http://dev.mysql.com/doc/refman/5.0/en/innodb-deadlocks.html
InnoDB uses automatic row-level locking. You can get deadlocks even in the case of transactions that just insert or delete a single row. That is because these operations are not really "atomic"; they automatically set locks on the (possibly several) index records of the row inserted or deleted.
- Parameters:
numRetries-maxRetries-transactionMode-conn-sql-params-- Returns:
- number of affected rows
- Since:
- 1.0.0
-
executeWithRetries
protected int executeWithRetries(int numRetries, int maxRetries, boolean transactionMode, Connection conn, String sql, Object... params)Execute a query, retry if deadlock.Note: http://dev.mysql.com/doc/refman/5.0/en/innodb-deadlocks.html
InnoDB uses automatic row-level locking. You can get deadlocks even in the case of transactions that just insert or delete a single row. That is because these operations are not really "atomic"; they automatically set locks on the (possibly several) index records of the row inserted or deleted.
- Parameters:
numRetries-maxRetries-transactionMode-conn-sql-params-- Returns:
- number of affected rows
- Since:
- 1.0.0
-
executeWithRetries
protected <T> T executeWithRetries(int numRetries, int maxRetries, Supplier<T> sqlRunner)Execute queries, retry if deadlock.Note: http://dev.mysql.com/doc/refman/5.0/en/innodb-deadlocks.html
InnoDB uses automatic row-level locking. You can get deadlocks even in the case of transactions that just insert or delete a single row. That is because these operations are not really "atomic"; they automatically set locks on the (possibly several) index records of the row inserted or deleted.
- Type Parameters:
T-- Parameters:
numRetries-maxRetries-sqlRunner-- Returns:
- Since:
- 1.0.0
-
_queueWithRetries
protected boolean _queueWithRetries(Connection conn, IQueueMessage<ID,DATA> immutableMsg, int numRetries, int maxRetries)
Queue a message, retry if deadlock.Note: http://dev.mysql.com/doc/refman/5.0/en/innodb-deadlocks.html
InnoDB uses automatic row-level locking. You can get deadlocks even in the case of transactions that just insert or delete a single row. That is because these operations are not really "atomic"; they automatically set locks on the (possibly several) index records of the row inserted or deleted.
- Parameters:
conn-immutableMsg-numRetries-maxRetries-- Returns:
-
_requeueSilentWithRetries
protected boolean _requeueSilentWithRetries(Connection conn, IQueueMessage<ID,DATA> immutableMsg, int numRetries, int maxRetries)
Re-queue (silentlt) a message, retry if deadlock.Note: http://dev.mysql.com/doc/refman/5.0/en/innodb-deadlocks.html
InnoDB uses automatic row-level locking. You can get deadlocks even in the case of transactions that just insert or delete a single row. That is because these operations are not really "atomic"; they automatically set locks on the (possibly several) index records of the row inserted or deleted.
- Parameters:
conn-immutableMsg-numRetries-maxRetries-- Returns:
-
_requeueWithRetries
protected boolean _requeueWithRetries(Connection conn, IQueueMessage<ID,DATA> immutableMsg, int numRetries, int maxRetries)
Re-queue a message, retry if deadlock.Note: http://dev.mysql.com/doc/refman/5.0/en/innodb-deadlocks.html
InnoDB uses automatic row-level locking. You can get deadlocks even in the case of transactions that just insert or delete a single row. That is because these operations are not really "atomic"; they automatically set locks on the (possibly several) index records of the row inserted or deleted.
- Parameters:
conn-immutableMsg-numRetries-maxRetries-- Returns:
-
doPutToQueue
protected boolean doPutToQueue(IQueueMessage<ID,DATA> msg, AbstractQueue.PutToQueueCase queueCase)
Put a message to the queue storage.Sub-class implements this method.
- Specified by:
doPutToQueuein classAbstractQueue<ID,DATA>- Returns:
-
_finishWithRetries
protected void _finishWithRetries(Connection conn, IQueueMessage<ID,DATA> msg, int numRetries, int maxRetries)
Perform "finish" action, retry if deadlock.Note: http://dev.mysql.com/doc/refman/5.0/en/innodb-deadlocks.html
InnoDB uses automatic row-level locking. You can get deadlocks even in the case of transactions that just insert or delete a single row. That is because these operations are not really "atomic"; they automatically set locks on the (possibly several) index records of the row inserted or deleted.
Note: the supplied queue message is mutable.
- Parameters:
conn-msg-numRetries-maxRetries-
-
finish
public void finish(IQueueMessage<ID,DATA> msg)
Called when finish processing the message to cleanup ephemeral storage.Implementation flow:
- Remove message from ephemeral storage.
Note: ephemeral storage implementation is optional, depends on implementation.
-
_takeWithRetries
protected IQueueMessage<ID,DATA> _takeWithRetries(Connection conn, int numRetries, int maxRetries)
Take a message from queue, retry if deadlock.Note: http://dev.mysql.com/doc/refman/5.0/en/innodb-deadlocks.html
InnoDB uses automatic row-level locking. You can get deadlocks even in the case of transactions that just insert or delete a single row. That is because these operations are not really "atomic"; they automatically set locks on the (possibly several) index records of the row inserted or deleted.
- Parameters:
conn-numRetries-maxRetries-- Returns:
-
take
public IQueueMessage<ID,DATA> take() throws QueueException.EphemeralIsFull
Take a message out of queue.Implementation flow:
- Read message from head of queue storage.
- Write message to ephemeral storage.
- Remove message from queue storage.
Note: ephemeral storage implementation is optional, depends on implementation.
- Returns:
- Throws:
QueueException.EphemeralIsFull- if the ephemeral storage is full
-
_getOrphanMessagesWithRetries
protected Collection<? extends IQueueMessage<ID,DATA>> _getOrphanMessagesWithRetries(long thresholdTimestampMs, Connection conn, int numRetries, int maxRetries)
Get all orphan messages (messages that were left in ephemeral storage for a long time), retry if deadlock.Note: http://dev.mysql.com/doc/refman/5.0/en/innodb-deadlocks.html
InnoDB uses automatic row-level locking. You can get deadlocks even in the case of transactions that just insert or delete a single row. That is because these operations are not really "atomic"; they automatically set locks on the (possibly several) index records of the row inserted or deleted.
- Parameters:
thresholdTimestampMs-conn-numRetries-maxRetries-- Returns:
- Since:
- 0.2.0
-
getOrphanMessages
public Collection<IQueueMessage<ID,DATA>> getOrphanMessages(long thresholdTimestampMs)
Get all orphan messages (messages that were left in ephemeral storage for a long time).- Parameters:
thresholdTimestampMs- message is orphan ifmessage's timestampMillis + thresholdTimestampMs < now. Which meansgetOrphanMessages(10000)will return orphan messages that have stayed in ephemeral storage for more than 10000 milliseconds.- Returns:
nullor empty collection if there is no orphan message
-
doSize
protected int doSize(Connection conn, String sql, String field, Object... params)
- Parameters:
conn-sql-field-params-- Returns:
- Since:
- 1.0.0
-
queueSize
protected int queueSize(Connection conn)
Get number of items currently in queue storage.- Parameters:
conn-- Returns:
- Since:
- 0.5.0
-
ephemeralSize
protected int ephemeralSize(Connection conn)
Get number of items currently in ephemeral storage.- Parameters:
conn-- Returns:
- Since:
- 0.5.0
-
queueSize
public int queueSize()
Get number of items currently in queue storage.- Returns:
- negative number if queue size can not be queried
-
ephemeralSize
public int ephemeralSize()
Get number of items currently in ephemeral storage.Note: ephemeral storage implementation is optional, depends on implementation.
- Returns:
- negative number if ephemeral size can not be queried
-
-