Package com.github.ambry.account

Class HelixAccountService

  • All Implemented Interfaces:
    com.github.ambry.account.AccountService, java.io.Closeable, java.lang.AutoCloseable
    public class HelixAccountService
extends java.lang.Object
implements com.github.ambry.account.AccountService

    An implementation of AccountService that employs a HelixPropertyStore as its underlying storage. It has two different AccountMetadataStore implementations. Use HelixAccountServiceConfig.useNewZNodePath to control which one to use. When the configuration is true, it uses the RouterStore. Otherwise, it uses LegacyMetadataStore. In both AccountMetadataStore, the latest full Account metadata will be cached locally, so serving an Account query will not incur any calls to remote ZooKeeper or AmbryServer.

    When a HelixAccountService starts up, it will automatically fetch a full set of Account metadata and store them in its local cache. It also takes a Notifier, and subscribes to a ACCOUNT_METADATA_CHANGE_TOPIC. Each time when receiving a FULL_ACCOUNT_METADATA_CHANGE_MESSAGE, it will fetch the updated full Account metadata, and refresh its local cache. After every successful operation for updating a collection of Accounts, it publishes a FULL_ACCOUNT_METADATA_CHANGE_MESSAGE for ACCOUNT_METADATA_CHANGE_TOPIC through the Notifier. If the remote Account metadata is corrupted or has conflict, HelixAccountService will not update its local cache.

    Previously, HelixAccountService only keeps a backup when there is an update Account HTTP request received by this instance. It doesn't backup mutations made by other instances. Since HTTP requests to update Account are rare, latest backup file often holds a out-of-date view of the Account metadata at lots of instances. In order to keep backup file up-to-date, in the new implementation, each mutation to the Account metadata will be persisted with BackupFileManager. This is achieved by the fact the HelixAccountService will publish each mutation to ZooKeeper and upon receiving the mutation message, all the HelixAccountService instances will fetch the latest Account metadata. And when it does, it also persists the latest Account metadata in the backup. Every time a mutation of account metadata is performed, a new backup file will be created for locally, in a directory specified by the HelixAccountServiceConfig.backupDir. A backup file is created when the helix listener on ACCOUNT_METADATA_CHANGE_TOPIC is notified. And backup's filename could contain version and last modified time information of the znode that stores the account metadata. The flow to update account looks like

    1. updateAccounts(Collection) is invoked
    2. AccountMetadataStore merges the incoming account mutation request with the current account set and write it back to helix
    3. Notifier publish this action to helix by writing to ACCOUNT_METADATA_CHANGE_TOPIC
    4. Listeners listening on ACCOUNT_METADATA_CHANGE_TOPIC fetches the new account metadata set
    5. Listeners persists this new set as a backup file through BackupFileManager
    There are limited number of backup files can be persisted. Once the number is exceeded, BackupFileManager would start removing the oldest backup file. The number can be configured through HelixAccountServiceConfig.maxBackupFileCount.

    The full set of the Account metadata are stored in a single ZNRecord in LegacyMetadataStore or a single blob in RouterStore, as a simple map from a string account id to the Account content in json as a string.

    Limited by HelixPropertyStore, the total size of Account data stored on a single ZNRecord cannot exceed 1MB. If the serialized json string exceeded 1MB limitation, it will then be compressed before saving back to zookeeper.

    • Field Summary

      Fields 
      Modifier and Type Field Description
      protected java.util.concurrent.atomic.AtomicReference<com.github.ambry.account.AccountInfoMap> accountInfoMapRef  
      protected AccountServiceMetrics accountServiceMetrics  
      protected java.util.concurrent.CopyOnWriteArraySet<java.util.function.Consumer<java.util.Collection<com.github.ambry.account.Account>>> accountUpdateConsumers  
      protected com.github.ambry.commons.TopicListener<java.lang.String> changeTopicListener  
      protected java.util.concurrent.locks.ReentrantLock lock  
      protected com.github.ambry.commons.Notifier<java.lang.String> notifier  

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      boolean addAccountUpdateConsumer​(java.util.function.Consumer<java.util.Collection<com.github.ambry.account.Account>> accountUpdateConsumer)  
      protected void checkOpen()
      Ensures the account service is ready to process requests.
      void close()  
      com.github.ambry.account.Account getAccountById​(short id)  
      com.github.ambry.account.Account getAccountByName​(java.lang.String accountName)  
      java.util.Collection<com.github.ambry.account.Account> getAllAccounts()  
      protected void maybeSubscribeChangeTopic​(boolean reportNull)  
      protected void maybeUnsubscribeChangeTopic()  
      protected void notifyAccountUpdateConsumers​(java.util.Collection<com.github.ambry.account.Account> updatedAccounts, boolean isCalledFromListener)
      Logs and notifies account update Consumers about any new account changes/creations.
      protected void onAccountChangeMessage​(java.lang.String topic, java.lang.String message)
      To be used to subscribe to a Notifier topic.
      protected void publishChangeNotice()  
      boolean removeAccountUpdateConsumer​(java.util.function.Consumer<java.util.Collection<com.github.ambry.account.Account>> accountUpdateConsumer)  
      void selectInactiveContainersAndMarkInStore​(com.github.ambry.server.StatsSnapshot statsSnapshot)
      Selects Containers to be marked as INACTIVE and marked in underlying account store.
      void setupRouter​(com.github.ambry.router.Router router)
      set the router to the given one.
      void updateAccounts​(java.util.Collection<com.github.ambry.account.Account> accounts)
      java.util.Collection<com.github.ambry.account.Container> updateContainers​(java.lang.String accountName, java.util.Collection<com.github.ambry.account.Container> containers)  
      protected void updateResolvedContainers​(com.github.ambry.account.Account account, java.util.Collection<com.github.ambry.account.Container> resolvedContainers)
      Update the containers that have been vetted as non-duplicates and populated with ids.

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

      • Methods inherited from interface com.github.ambry.account.AccountService

        addAccountUpdateConsumer, getAccountById, getAccountByName, getAllAccounts, getContainerById, getContainerByName, getContainersByStatus, removeAccountUpdateConsumer, selectInactiveContainersAndMarkInStore, updateContainers

    • Field Detail

      • accountInfoMapRef

        protected final java.util.concurrent.atomic.AtomicReference<com.github.ambry.account.AccountInfoMap> accountInfoMapRef

      • lock

        protected final java.util.concurrent.locks.ReentrantLock lock

      • accountUpdateConsumers

        protected final java.util.concurrent.CopyOnWriteArraySet<java.util.function.Consumer<java.util.Collection<com.github.ambry.account.Account>>> accountUpdateConsumers

      • notifier

        protected final com.github.ambry.commons.Notifier<java.lang.String> notifier

      • changeTopicListener

        protected final com.github.ambry.commons.TopicListener<java.lang.String> changeTopicListener

    • Method Detail

      • setupRouter

        public void setupRouter​(com.github.ambry.router.Router router)
                 throws java.lang.IllegalStateException
        set the router to the given one. This is a blocking call. This function would block until it fetches the Account metadata from ambry server.
        Parameters:
        router - The router to set.
        Throws:
        java.lang.IllegalStateException - when the router already set up.

      • updateAccounts

        public void updateAccounts​(java.util.Collection<com.github.ambry.account.Account> accounts)
                    throws com.github.ambry.account.AccountServiceException

        This call is blocking until it completes the operation of updating account metadata to HelixPropertyStore.

        There is a slight chance that an Account could be updated successfully, but its value was set based on an outdated Account. This can be fixed after generationId is introduced to Account.

        Specified by:
        updateAccounts in interface com.github.ambry.account.AccountService
        Throws:
        com.github.ambry.account.AccountServiceException

      • close

        public void close()
        Specified by:
        close in interface java.lang.AutoCloseable
        Specified by:
        close in interface java.io.Closeable

      • onAccountChangeMessage

        protected void onAccountChangeMessage​(java.lang.String topic,
                                      java.lang.String message)
        To be used to subscribe to a Notifier topic. Upon receiving a FULL_ACCOUNT_METADATA_CHANGE_MESSAGE, it will check for any account updates.
        Parameters:
        topic - The topic.
        message - The message for the topic.

      • checkOpen

        protected void checkOpen()
        Ensures the account service is ready to process requests. Throws an unchecked exception (for example IllegalStateException) if the account service is not ready to process requests.

      • maybeSubscribeChangeTopic

        protected void maybeSubscribeChangeTopic​(boolean reportNull)

      • maybeUnsubscribeChangeTopic

        protected void maybeUnsubscribeChangeTopic()

      • publishChangeNotice

        protected void publishChangeNotice()

      • getAccountByName

        public com.github.ambry.account.Account getAccountByName​(java.lang.String accountName)
        Specified by:
        getAccountByName in interface com.github.ambry.account.AccountService

      • getAccountById

        public com.github.ambry.account.Account getAccountById​(short id)
        Specified by:
        getAccountById in interface com.github.ambry.account.AccountService

      • getAllAccounts

        public java.util.Collection<com.github.ambry.account.Account> getAllAccounts()
        Specified by:
        getAllAccounts in interface com.github.ambry.account.AccountService

      • updateContainers

        public java.util.Collection<com.github.ambry.account.Container> updateContainers​(java.lang.String accountName,
                                                                                 java.util.Collection<com.github.ambry.account.Container> containers)
                                                                          throws com.github.ambry.account.AccountServiceException
        Specified by:
        updateContainers in interface com.github.ambry.account.AccountService
        Throws:
        com.github.ambry.account.AccountServiceException

      • updateResolvedContainers

        protected void updateResolvedContainers​(com.github.ambry.account.Account account,
                                        java.util.Collection<com.github.ambry.account.Container> resolvedContainers)
                                 throws com.github.ambry.account.AccountServiceException
        Update the containers that have been vetted as non-duplicates and populated with ids.
        Parameters:
        account - the account owning the containers.
        resolvedContainers - the resolved containers.
        Throws:
        com.github.ambry.account.AccountServiceException

      • addAccountUpdateConsumer

        public boolean addAccountUpdateConsumer​(java.util.function.Consumer<java.util.Collection<com.github.ambry.account.Account>> accountUpdateConsumer)
        Specified by:
        addAccountUpdateConsumer in interface com.github.ambry.account.AccountService

      • removeAccountUpdateConsumer

        public boolean removeAccountUpdateConsumer​(java.util.function.Consumer<java.util.Collection<com.github.ambry.account.Account>> accountUpdateConsumer)
        Specified by:
        removeAccountUpdateConsumer in interface com.github.ambry.account.AccountService

      • notifyAccountUpdateConsumers

        protected void notifyAccountUpdateConsumers​(java.util.Collection<com.github.ambry.account.Account> updatedAccounts,
                                            boolean isCalledFromListener)
        Logs and notifies account update Consumers about any new account changes/creations.
        Parameters:
        updatedAccounts - collection of updated accounts
        isCalledFromListener - true if the caller is the account update listener, false otherwise.

      • selectInactiveContainersAndMarkInStore

        public void selectInactiveContainersAndMarkInStore​(com.github.ambry.server.StatsSnapshot statsSnapshot)
        Selects Containers to be marked as INACTIVE and marked in underlying account store.
        Specified by:
        selectInactiveContainersAndMarkInStore in interface com.github.ambry.account.AccountService