Package com.github.ambry.account

Class HelixAccountService

  • All Implemented Interfaces:
    com.github.ambry.account.AccountService, java.io.Closeable, java.lang.AutoCloseable
    public class HelixAccountService
extends java.lang.Object
implements com.github.ambry.account.AccountService

    An implementation of AccountService that employs a HelixPropertyStore as its underlying storage. It has two different AccountMetadataStore implementations. Use HelixAccountServiceConfig.useNewZNodePath to control which one to use. When the configuration is true, it uses the RouterStore. Otherwise, it uses LegacyMetadataStore. In both AccountMetadataStore, the latest full Account metadata will be cached locally, so serving an Account query will not incur any calls to remote ZooKeeper or AmbryServer.

    When a HelixAccountService starts up, it will automatically fetch a full set of Account metadata and store them in its local cache. It also takes a Notifier, and subscribes to a ACCOUNT_METADATA_CHANGE_TOPIC. Each time when receiving a FULL_ACCOUNT_METADATA_CHANGE_MESSAGE, it will fetch the updated full Account metadata, and refresh its local cache. After every successful operation for updating a collection of Accounts, it publishes a FULL_ACCOUNT_METADATA_CHANGE_MESSAGE for ACCOUNT_METADATA_CHANGE_TOPIC through the Notifier. If the remote Account metadata is corrupted or has conflict, HelixAccountService will not update its local cache.

    Previously, HelixAccountService only keeps a backup when there is an update Account HTTP request received by this instance. It doesn't backup mutations made by other instances. Since HTTP requests to update Account are rare, latest backup file often holds a out-of-date view of the Account metadata at lots of instances. In order to keep backup file up-to-date, in the new implementation, each mutation to the Account metadata will be persisted with BackupFileManager. This is achieved by the fact the HelixAccountService will publish each mutation to ZooKeeper and upon receiving the mutation message, all the HelixAccountService instances will fetch the latest Account metadata. And when it does, it also persists the latest Account metadata in the backup. Every time a mutation of account metadata is performed, a new backup file will be created for locally, in a directory specified by the HelixAccountServiceConfig.backupDir. A backup file is created when the helix listener on ACCOUNT_METADATA_CHANGE_TOPIC is notified. And backup's filename could contain version and last modified time information of the znode that stores the account metadata. The flow to update account looks like

    1. updateAccounts(Collection) is invoked
    2. AccountMetadataStore merges the incoming account mutation request with the current account set and write it back to helix
    3. Notifier publish this action to helix by writing to ACCOUNT_METADATA_CHANGE_TOPIC
    4. Listeners listening on ACCOUNT_METADATA_CHANGE_TOPIC fetches the new account metadata set
    5. Listeners persists this new set as a backup file through BackupFileManager
    There are limited number of backup files can be persisted. Once the number is exceeded, BackupFileManager would start removing the oldest backup file. The number can be configured through HelixAccountServiceConfig.maxBackupFileCount.

    The full set of the Account metadata are stored in a single ZNRecord in LegacyMetadataStore or a single blob in RouterStore, as a simple map from a string account id to the Account content in json as a string.

    Limited by HelixPropertyStore, the total size of Account data stored on a single ZNRecord cannot exceed 1MB. If the serialized json string exceeded 1MB limitation, it will then be compressed before saving back to zookeeper.

    • Field Summary

      Fields 
      Modifier and Type Field Description
      protected java.util.concurrent.atomic.AtomicReference<com.github.ambry.account.AccountInfoMap> accountInfoMapRef  
      protected AccountServiceMetrics accountServiceMetrics  
      protected java.util.concurrent.CopyOnWriteArraySet<java.util.function.Consumer<java.util.Collection<com.github.ambry.account.Account>>> accountUpdateConsumers  
      protected java.util.concurrent.locks.ReentrantLock lock  

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      boolean addAccountUpdateConsumer​(java.util.function.Consumer<java.util.Collection<com.github.ambry.account.Account>> accountUpdateConsumer)  
      protected void checkOpen()
      Ensures the account service is ready to process requests.
      void close()  
      com.github.ambry.account.Account getAccountById​(short id)  
      com.github.ambry.account.Account getAccountByName​(java.lang.String accountName)  
      java.util.Collection<com.github.ambry.account.Account> getAllAccounts()  
      protected void notifyAccountUpdateConsumers​(com.github.ambry.account.AccountInfoMap newAccountInfoMap, com.github.ambry.account.AccountInfoMap oldAccountInfoMap, boolean isCalledFromListener)
      Logs and notifies account update Consumers about any new account changes/creations.
      boolean removeAccountUpdateConsumer​(java.util.function.Consumer<java.util.Collection<com.github.ambry.account.Account>> accountUpdateConsumer)  
      void selectInactiveContainersAndMarkInZK​(com.github.ambry.server.StatsSnapshot statsSnapshot)
      Selects Containers to be marked as INACTIVE and marked in zookeeper.
      void setupRouter​(com.github.ambry.router.Router router)
      set the router to the given one.
      boolean updateAccounts​(java.util.Collection<com.github.ambry.account.Account> accounts)
      java.util.Collection<com.github.ambry.account.Container> updateContainers​(java.lang.String accountName, java.util.Collection<com.github.ambry.account.Container> containers)  

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

      • Methods inherited from interface com.github.ambry.account.AccountService

        addAccountUpdateConsumer, getAccountById, getAccountByName, getAllAccounts, getContainer, getContainersByStatus, removeAccountUpdateConsumer, updateContainers

    • Field Detail

      • accountInfoMapRef

        protected final java.util.concurrent.atomic.AtomicReference<com.github.ambry.account.AccountInfoMap> accountInfoMapRef

      • lock

        protected final java.util.concurrent.locks.ReentrantLock lock

      • accountUpdateConsumers

        protected final java.util.concurrent.CopyOnWriteArraySet<java.util.function.Consumer<java.util.Collection<com.github.ambry.account.Account>>> accountUpdateConsumers

    • Method Detail

      • setupRouter

        public void setupRouter​(com.github.ambry.router.Router router)
                 throws java.lang.IllegalStateException
        set the router to the given one. This is a blocking call. This function would block until it fetches the Account metadata from ambry server.
        Parameters:
        router - The router to set.
        Throws:
        java.lang.IllegalStateException - when the router already set up.

      • updateAccounts

        public boolean updateAccounts​(java.util.Collection<com.github.ambry.account.Account> accounts)

        This call is blocking until it completes the operation of updating account metadata to HelixPropertyStore.

        There is a slight chance that an Account could be updated successfully, but its value was set based on an outdated Account. This can be fixed after generationId is introduced to Account.

        Specified by:
        updateAccounts in interface com.github.ambry.account.AccountService

      • close

        public void close()
        Specified by:
        close in interface java.lang.AutoCloseable
        Specified by:
        close in interface java.io.Closeable

      • selectInactiveContainersAndMarkInZK

        public void selectInactiveContainersAndMarkInZK​(com.github.ambry.server.StatsSnapshot statsSnapshot)
        Selects Containers to be marked as INACTIVE and marked in zookeeper.
        Specified by:
        selectInactiveContainersAndMarkInZK in interface com.github.ambry.account.AccountService

      • checkOpen

        protected void checkOpen()
        Ensures the account service is ready to process requests. Throws an unchecked exception (for example IllegalStateException) if the account service is not ready to process requests.

      • getAccountByName

        public com.github.ambry.account.Account getAccountByName​(java.lang.String accountName)
        Specified by:
        getAccountByName in interface com.github.ambry.account.AccountService

      • getAccountById

        public com.github.ambry.account.Account getAccountById​(short id)
        Specified by:
        getAccountById in interface com.github.ambry.account.AccountService

      • getAllAccounts

        public java.util.Collection<com.github.ambry.account.Account> getAllAccounts()
        Specified by:
        getAllAccounts in interface com.github.ambry.account.AccountService

      • updateContainers

        public java.util.Collection<com.github.ambry.account.Container> updateContainers​(java.lang.String accountName,
                                                                                 java.util.Collection<com.github.ambry.account.Container> containers)
                                                                          throws com.github.ambry.account.AccountServiceException
        Specified by:
        updateContainers in interface com.github.ambry.account.AccountService
        Throws:
        com.github.ambry.account.AccountServiceException

      • addAccountUpdateConsumer

        public boolean addAccountUpdateConsumer​(java.util.function.Consumer<java.util.Collection<com.github.ambry.account.Account>> accountUpdateConsumer)
        Specified by:
        addAccountUpdateConsumer in interface com.github.ambry.account.AccountService

      • removeAccountUpdateConsumer

        public boolean removeAccountUpdateConsumer​(java.util.function.Consumer<java.util.Collection<com.github.ambry.account.Account>> accountUpdateConsumer)
        Specified by:
        removeAccountUpdateConsumer in interface com.github.ambry.account.AccountService

      • notifyAccountUpdateConsumers

        protected void notifyAccountUpdateConsumers​(com.github.ambry.account.AccountInfoMap newAccountInfoMap,
                                            com.github.ambry.account.AccountInfoMap oldAccountInfoMap,
                                            boolean isCalledFromListener)
        Logs and notifies account update Consumers about any new account changes/creations.
        Parameters:
        newAccountInfoMap - the new AccountInfoMap that has been set.
        oldAccountInfoMap - the AccountInfoMap that was cached before this change.
        isCalledFromListener - true if the caller is the account update listener, {@@code false} otherwise.