Package org.apache.pulsar.functions.worker

Class FunctionMetaDataManager

java.lang.Object
org.apache.pulsar.functions.worker.FunctionMetaDataManager
All Implemented Interfaces:
AutoCloseable
public class FunctionMetaDataManager extends Object implements AutoCloseable
FunctionMetaDataManager maintains a global state of all function metadata. It is the system of record for the worker for function metadata. FunctionMetaDataManager operates in either the leader mode or worker mode. By default, when you initialize and start manager, it starts in the worker mode. In the worker mode, the FunctionMetaDataTailer tails the function metadata topic and updates the in-memory metadata cache. When the worker becomes a leader, it calls the acquireLeadaership thru which the FunctionMetaData Manager switches to a leader mode. In the leader mode the manager first captures an exclusive producer on the the metadata topic. Then it drains the MetaDataTailer to ensure that it has caught up to the last record. After this point, the worker can update the in-memory state of function metadata by calling processUpdate/processDeregister methods. If a worker loses its leadership, it calls giveupLeaderShip at which time the manager closes its exclusive producer and starts its tailer again.

  • Constructor Details

    • FunctionMetaDataManager

      public FunctionMetaDataManager(org.apache.pulsar.functions.worker.WorkerConfig workerConfig, SchedulerManager schedulerManager, org.apache.pulsar.client.api.PulsarClient pulsarClient, ErrorNotifier errorNotifier) throws org.apache.pulsar.client.api.PulsarClientException
      Throws:
      org.apache.pulsar.client.api.PulsarClientException

  • Method Details

    • initialize

      public void initialize()
      Initializes the FunctionMetaDataManager. We create a new reader

    • start

      public void start()

    • close

      public void close() throws Exception
      Specified by:
      close in interface AutoCloseable
      Throws:
      Exception

    • getFunctionMetaData

      public org.apache.pulsar.functions.proto.Function.FunctionMetaData getFunctionMetaData(String tenant, String namespace, String functionName)
      Get the function metadata for a function.
      Parameters:
      tenant - the tenant the function belongs to
      namespace - the namespace the function belongs to
      functionName - the function name
      Returns:
      FunctionMetaData that contains the function metadata

    • getAllFunctionMetaData

      public List<org.apache.pulsar.functions.proto.Function.FunctionMetaData> getAllFunctionMetaData()
      Get a list of all the meta for every function.
      Returns:
      list of function metadata

    • listFunctions

      public Collection<org.apache.pulsar.functions.proto.Function.FunctionMetaData> listFunctions(String tenant, String namespace)
      List all the functions in a namespace.
      Parameters:
      tenant - the tenant the namespace belongs to
      namespace - the namespace
      Returns:
      a list of function names

    • containsFunction

      public boolean containsFunction(String tenant, String namespace, String functionName)
      Check if the function exists.
      Parameters:
      tenant - tenant that the function belongs to
      namespace - namespace that the function belongs to
      functionName - name of function
      Returns:
      true if function exists and false if it does not

    • updateFunctionOnLeader

      public void updateFunctionOnLeader(org.apache.pulsar.functions.proto.Function.FunctionMetaData functionMetaData, boolean delete) throws IllegalStateException, IllegalArgumentException
      Called by the worker when we are in the leader mode. In this state, we update our in-memory data structures and then write to the metadata topic.
      Parameters:
      functionMetaData - The function metadata in question
      delete - Is this a delete operation
      Throws:
      IllegalStateException - if we are not the leader
      IllegalArgumentException - if the request is out of date.

    • acquireExclusiveWrite

      public org.apache.pulsar.client.api.Producer<byte[]> acquireExclusiveWrite(Supplier<Boolean> isLeader) throws WorkerUtils.NotLeaderAnymore
      Acquires a exclusive producer. This method cannot return null. It can only return a valid exclusive producer or throw NotLeaderAnymore exception.
      Parameters:
      isLeader - if the worker is still the leader
      Returns:
      A valid exclusive producer
      Throws:
      WorkerUtils.NotLeaderAnymore - if the worker is no longer the leader.

    • acquireLeadership

      public void acquireLeadership(org.apache.pulsar.client.api.Producer<byte[]> exclusiveProducer)
      Called by the leader service when this worker becomes the leader. We first get exclusive producer on the metadata topic. Next we drain the tailer to ensure that we have caught up to metadata topic. After which we close the tailer. Note that this method cannot be syncrhonized because the tailer might still be processing messages

    • giveupLeadership

      public void giveupLeadership()
      called by the leader service when we lose leadership. We close the exclusive producer and start the tailer.

    • processMetaDataTopicMessage

      public void processMetaDataTopicMessage(org.apache.pulsar.client.api.Message<byte[]> message) throws IOException
      This is called by the MetaData tailer. It updates the in-memory cache. It eats up any exception thrown by processUpdate/processDeregister since that's just part of the state machine
      Parameters:
      message - The message read from metadata topic that needs to be processed
      Throws:
      IOException