Package org.apache.pulsar.functions.worker

Class FunctionMetaDataManager

  • All Implemented Interfaces:
    java.lang.AutoCloseable
    public class FunctionMetaDataManager
extends java.lang.Object
implements java.lang.AutoCloseable
    FunctionMetaDataManager maintains a global state of all function metadata. It is the system of record for the worker for function metadata. FunctionMetaDataManager operates in either the leader mode or worker mode. By default, when you initialize and start manager, it starts in the worker mode. In the worker mode, the FunctionMetaDataTailer tails the function metadata topic and updates the in-memory metadata cache. When the worker becomes a leader, it calls the acquireLeadaership thru which the FunctionMetaData Manager switches to a leader mode. In the leader mode the manager first captures an exclusive producer on the the metadata topic. Then it drains the MetaDataTailer to ensure that it has caught up to the last record. After this point, the worker can update the in-memory state of function metadata by calling processUpdate/processDeregister methods. If a worker loses its leadership, it calls giveupLeaderShip at which time the manager closes its exclusive producer and starts its tailer again.

    • Constructor Summary

      Constructors 
      Constructor Description
      FunctionMetaDataManager​(org.apache.pulsar.functions.worker.WorkerConfig workerConfig, SchedulerManager schedulerManager, org.apache.pulsar.client.api.PulsarClient pulsarClient, ErrorNotifier errorNotifier)  

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      org.apache.pulsar.client.api.Producer<byte[]> acquireExclusiveWrite​(java.util.function.Supplier<java.lang.Boolean> isLeader)
      Acquires a exclusive producer.
      void acquireLeadership​(org.apache.pulsar.client.api.Producer<byte[]> exclusiveProducer)
      Called by the leader service when this worker becomes the leader.
      void close()  
      boolean containsFunction​(java.lang.String tenant, java.lang.String namespace, java.lang.String functionName)
      Check if the function exists
      java.util.List<org.apache.pulsar.functions.proto.Function.FunctionMetaData> getAllFunctionMetaData()
      Get a list of all the meta for every function
      org.apache.pulsar.functions.proto.Function.FunctionMetaData getFunctionMetaData​(java.lang.String tenant, java.lang.String namespace, java.lang.String functionName)
      Get the function metadata for a function
      void giveupLeadership()
      called by the leader service when we lose leadership.
      void initialize()
      Initializes the FunctionMetaDataManager.
      java.util.Collection<org.apache.pulsar.functions.proto.Function.FunctionMetaData> listFunctions​(java.lang.String tenant, java.lang.String namespace)
      List all the functions in a namespace
      void processMetaDataTopicMessage​(org.apache.pulsar.client.api.Message<byte[]> message)
      This is called by the MetaData tailer.
      void start()  
      void updateFunctionOnLeader​(org.apache.pulsar.functions.proto.Function.FunctionMetaData functionMetaData, boolean delete)
      Called by the worker when we are in the leader mode.

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Constructor Detail

      • FunctionMetaDataManager

        public FunctionMetaDataManager​(org.apache.pulsar.functions.worker.WorkerConfig workerConfig,
                               SchedulerManager schedulerManager,
                               org.apache.pulsar.client.api.PulsarClient pulsarClient,
                               ErrorNotifier errorNotifier)
                        throws org.apache.pulsar.client.api.PulsarClientException
        Throws:
        org.apache.pulsar.client.api.PulsarClientException

    • Method Detail

      • initialize

        public void initialize()
        Initializes the FunctionMetaDataManager. We create a new reader

      • start

        public void start()

      • close

        public void close()
           throws java.lang.Exception
        Specified by:
        close in interface java.lang.AutoCloseable
        Throws:
        java.lang.Exception

      • getFunctionMetaData

        public org.apache.pulsar.functions.proto.Function.FunctionMetaData getFunctionMetaData​(java.lang.String tenant,
                                                                                       java.lang.String namespace,
                                                                                       java.lang.String functionName)
        Get the function metadata for a function
        Parameters:
        tenant - the tenant the function belongs to
        namespace - the namespace the function belongs to
        functionName - the function name
        Returns:
        FunctionMetaData that contains the function metadata

      • getAllFunctionMetaData

        public java.util.List<org.apache.pulsar.functions.proto.Function.FunctionMetaData> getAllFunctionMetaData()
        Get a list of all the meta for every function
        Returns:
        list of function metadata

      • listFunctions

        public java.util.Collection<org.apache.pulsar.functions.proto.Function.FunctionMetaData> listFunctions​(java.lang.String tenant,
                                                                                                       java.lang.String namespace)
        List all the functions in a namespace
        Parameters:
        tenant - the tenant the namespace belongs to
        namespace - the namespace
        Returns:
        a list of function names

      • containsFunction

        public boolean containsFunction​(java.lang.String tenant,
                                java.lang.String namespace,
                                java.lang.String functionName)
        Check if the function exists
        Parameters:
        tenant - tenant that the function belongs to
        namespace - namespace that the function belongs to
        functionName - name of function
        Returns:
        true if function exists and false if it does not

      • updateFunctionOnLeader

        public void updateFunctionOnLeader​(org.apache.pulsar.functions.proto.Function.FunctionMetaData functionMetaData,
                                   boolean delete)
                            throws java.lang.IllegalStateException,
                                   java.lang.IllegalArgumentException
        Called by the worker when we are in the leader mode. In this state, we update our in-memory data structures and then write to the metadata topic.
        Parameters:
        functionMetaData - The function metadata in question
        delete - Is this a delete operation
        Throws:
        java.lang.IllegalStateException - if we are not the leader
        java.lang.IllegalArgumentException - if the request is out of date.

      • acquireExclusiveWrite

        public org.apache.pulsar.client.api.Producer<byte[]> acquireExclusiveWrite​(java.util.function.Supplier<java.lang.Boolean> isLeader)
                                                                    throws WorkerUtils.NotLeaderAnymore
        Acquires a exclusive producer. This method cannot return null. It can only return a valid exclusive producer or throw NotLeaderAnymore exception.
        Parameters:
        isLeader - if the worker is still the leader
        Returns:
        A valid exclusive producer
        Throws:
        WorkerUtils.NotLeaderAnymore - if the worker is no longer the leader.

      • acquireLeadership

        public void acquireLeadership​(org.apache.pulsar.client.api.Producer<byte[]> exclusiveProducer)
        Called by the leader service when this worker becomes the leader. We first get exclusive producer on the metadata topic. Next we drain the tailer to ensure that we have caught up to metadata topic. After which we close the tailer. Note that this method cannot be syncrhonized because the tailer might still be processing messages

      • giveupLeadership

        public void giveupLeadership()
        called by the leader service when we lose leadership. We close the exclusive producer and start the tailer.

      • processMetaDataTopicMessage

        public void processMetaDataTopicMessage​(org.apache.pulsar.client.api.Message<byte[]> message)
                                 throws java.io.IOException
        This is called by the MetaData tailer. It updates the in-memory cache. It eats up any exception thrown by processUpdate/processDeregister since that's just part of the state machine
        Parameters:
        message - The message read from metadata topic that needs to be processed
        Throws:
        java.io.IOException