Interface Repository

  • All Known Implementing Classes:
    AbstractRepository, JarRepository, LocalRepository, RemoteRepository, SimpleRepository, SimpleUrlRepository

    public interface Repository
    Repository is a format for storing data Artifacts for various uses including deep learning models and datasets.

    This repository format is based off of the design of the Maven Repository format (See maven). Unlike in Maven, the data doesn't need to be located within the repository. Instead, the repository only stores metadata including the URL and checksum of the actual data. When the artifact is prepared, the data is downloaded, checked, and then stored in the ~/.djl.ai/cache folder.

    The artifacts are first divided into a number of Metadata files that can each have multiple artifacts. The metadata files are identified by an MRL which contains:

    • type - The resource type, e.g. model or dataset.
    • Application - The resource application (See Application).
    • Group Id - The group id identifies the group publishing the artifacts using a reverse domain name system.
    • Artifact Id - The artifact id identifies the different artifacts published by a single group.

    Within each metadata are a number of artifacts that share the same groupId, artifactId, name, description, website, and update date. The artifacts within the metadata differ primarily based on name and properties. Note that there is a metadata name and a separate artifact name. The properties are a map with string property names and string property values that can be used to represent key differentiators between artifacts such as dataset, flavors, and image sizes. For example, you might have a ResNet metadata file with different artifacts to represent different hyperparameters and datasets used for training the ResNet.

    Each artifact contains a Version number (which can be a snapshot version). The data in the artifacts are represented by files in the format of an Artifact.Item and a parsed JSON object of arguments. The files can either by a single file, an automatically extracted gzip file, or an automatically extracted zip file that will be treated as a directory. These can be used to store data such as the dataset, model parameters, and synset files. The arguments can be used to store data about the model used for initialization. For example, it can store the image size which can be used by the model loader for both initializing the block and setting up resizing in the translator.

    There are three kinds of repositories: a LocalRepository, RemoteRepository, and SimpleRepository. For all three kinds, new repositories should be created by calling newInstance(String, String) with the location of the repository.

    • Method Summary

      All Methods Static Methods Instance Methods Abstract Methods Default Methods 
      Modifier and Type Method Description
      void addResource​(MRL mrl)
      Adds resource to the repository.
      default MRL dataset​(Application application, java.lang.String groupId, java.lang.String artifactId)
      Creates a dataset MRL with specified application.
      default MRL dataset​(Application application, java.lang.String groupId, java.lang.String artifactId, java.lang.String version)
      Creates a dataset MRL with specified application.
      java.net.URI getBaseUri()
      Returns the URI to the base of the repository.
      java.nio.file.Path getCacheDirectory()
      Returns the cache directory for the repository.
      java.nio.file.Path getFile​(Artifact.Item item, java.lang.String path)
      Returns the path to a file for the item.
      java.lang.String getName()
      Returns the repository name.
      default java.nio.file.Path getResourceDirectory​(Artifact artifact)
      Returns the resource directory for the an artifact.
      java.util.List<MRL> getResources()
      Returns a list of MRLs in the repository.
      boolean isRemote()
      Returns whether the repository is remote repository.
      java.lang.String[] listDirectory​(Artifact.Item item, java.lang.String path)
      Returns the list of files directly within a specified directory in a zipped directory item.
      Metadata locate​(MRL mrl)
      Returns the metadata at a mrl.
      default MRL model​(Application application, java.lang.String groupId, java.lang.String artifactId)
      Creates a model MRL with specified application.
      default MRL model​(Application application, java.lang.String groupId, java.lang.String artifactId, java.lang.String version)
      Creates a model MRL with specified application.
      default MRL model​(Application application, java.lang.String groupId, java.lang.String artifactId, java.lang.String version, java.lang.String artifactName)
      Creates a model MRL with specified application.
      static Repository newInstance​(java.lang.String name, java.lang.String url)
      Creates a new instance of a repository with a name and url.
      static Repository newInstance​(java.lang.String name, java.nio.file.Path path)
      Creates a new instance of a repository with a name and url.
      java.io.InputStream openStream​(Artifact.Item item, java.lang.String path)
      Returns an InputStream for an item in a repository.
      default void prepare​(Artifact artifact)
      Prepares the artifact for use.
      void prepare​(Artifact artifact, ai.djl.util.Progress progress)
      Prepares the artifact for use with progress tracking.
      static void registerRepositoryFactory​(RepositoryFactory factory)
      Registers a RepositoryFactory to handle the specified url scheme.
      Artifact resolve​(MRL mrl, java.util.Map<java.lang.String,​java.lang.String> filter)
      Returns the artifact matching a mrl, version, and property filter.
    • Method Detail

      • newInstance

        static Repository newInstance​(java.lang.String name,
                                      java.nio.file.Path path)
        Creates a new instance of a repository with a name and url.
        Parameters:
        name - the repository name
        path - the repository location
        Returns:
        the new repository
      • newInstance

        static Repository newInstance​(java.lang.String name,
                                      java.lang.String url)
        Creates a new instance of a repository with a name and url.
        Parameters:
        name - the repository name
        url - the repository location
        Returns:
        the new repository
      • model

        default MRL model​(Application application,
                          java.lang.String groupId,
                          java.lang.String artifactId)
        Creates a model MRL with specified application.
        Parameters:
        application - the desired application
        groupId - the desired groupId
        artifactId - the desired artifactId
        Returns:
        a model MRL
      • model

        default MRL model​(Application application,
                          java.lang.String groupId,
                          java.lang.String artifactId,
                          java.lang.String version)
        Creates a model MRL with specified application.
        Parameters:
        application - the desired application
        groupId - the desired groupId
        artifactId - the desired artifactId
        version - the resource version
        Returns:
        a model MRL
      • model

        default MRL model​(Application application,
                          java.lang.String groupId,
                          java.lang.String artifactId,
                          java.lang.String version,
                          java.lang.String artifactName)
        Creates a model MRL with specified application.
        Parameters:
        application - the desired application
        groupId - the desired groupId
        artifactId - the desired artifactId
        version - the resource version
        artifactName - the desired artifact name
        Returns:
        a model MRL
      • dataset

        default MRL dataset​(Application application,
                            java.lang.String groupId,
                            java.lang.String artifactId)
        Creates a dataset MRL with specified application.
        Parameters:
        application - the desired application
        groupId - the desired groupId
        artifactId - the desired artifactId
        Returns:
        a dataset MRL
      • dataset

        default MRL dataset​(Application application,
                            java.lang.String groupId,
                            java.lang.String artifactId,
                            java.lang.String version)
        Creates a dataset MRL with specified application.
        Parameters:
        application - the desired application
        groupId - the desired groupId
        artifactId - the desired artifactId
        version - the resource version
        Returns:
        a dataset MRL
      • isRemote

        boolean isRemote()
        Returns whether the repository is remote repository.
        Returns:
        whether the repository is remote repository
      • getName

        java.lang.String getName()
        Returns the repository name.
        Returns:
        the repository name
      • getBaseUri

        java.net.URI getBaseUri()
        Returns the URI to the base of the repository.
        Returns:
        the URI
      • locate

        Metadata locate​(MRL mrl)
                 throws java.io.IOException
        Returns the metadata at a mrl.
        Parameters:
        mrl - the mrl of the metadata to retrieve
        Returns:
        the metadata
        Throws:
        java.io.IOException - if it failed to load the metadata
      • resolve

        Artifact resolve​(MRL mrl,
                         java.util.Map<java.lang.String,​java.lang.String> filter)
                  throws java.io.IOException
        Returns the artifact matching a mrl, version, and property filter.
        Parameters:
        mrl - the mrl to match the artifact against
        filter - the property filter
        Returns:
        the matched artifact
        Throws:
        java.io.IOException - if it failed to load the artifact
      • openStream

        java.io.InputStream openStream​(Artifact.Item item,
                                       java.lang.String path)
                                throws java.io.IOException
        Returns an InputStream for an item in a repository.
        Parameters:
        item - the item to open
        path - the path to a file if the item is a zipped directory. Otherwise, pass null
        Returns:
        the file stream
        Throws:
        java.io.IOException - if it failed to open the stream
      • getFile

        java.nio.file.Path getFile​(Artifact.Item item,
                                   java.lang.String path)
                            throws java.io.IOException
        Returns the path to a file for the item.
        Parameters:
        item - the item to find the path for
        path - the path to a file if the item is a zipped directory. Otherwise, pass null
        Returns:
        the file path
        Throws:
        java.io.IOException - if it failed to find the path
      • listDirectory

        java.lang.String[] listDirectory​(Artifact.Item item,
                                         java.lang.String path)
                                  throws java.io.IOException
        Returns the list of files directly within a specified directory in a zipped directory item.
        Parameters:
        item - the zipped directory item
        path - the path within the zip directory
        Returns:
        the list of files/directories
        Throws:
        java.io.IOException - if it failed to list the directory
      • prepare

        default void prepare​(Artifact artifact)
                      throws java.io.IOException
        Prepares the artifact for use.
        Parameters:
        artifact - the artifact to prepare
        Throws:
        java.io.IOException - if it failed to prepare
      • prepare

        void prepare​(Artifact artifact,
                     ai.djl.util.Progress progress)
              throws java.io.IOException
        Prepares the artifact for use with progress tracking.
        Parameters:
        artifact - the artifact to prepare
        progress - the progress tracker
        Throws:
        java.io.IOException - if it failed to prepare
      • getCacheDirectory

        java.nio.file.Path getCacheDirectory()
                                      throws java.io.IOException
        Returns the cache directory for the repository.
        Returns:
        the cache directory path
        Throws:
        java.io.IOException - if it failed to ensure the creation of the cache directory
      • getResourceDirectory

        default java.nio.file.Path getResourceDirectory​(Artifact artifact)
                                                 throws java.io.IOException
        Returns the resource directory for the an artifact.
        Parameters:
        artifact - the artifact whose resource directory to return
        Returns:
        the resource directory path
        Throws:
        java.io.IOException - if it failed to ensure the creation of the cache directory
      • getResources

        java.util.List<MRL> getResources()
        Returns a list of MRLs in the repository.

        An empty list will be returned if underlying Repository implementation does not support this feature.

        Returns:
        a list of MRLs in the repository
      • addResource

        void addResource​(MRL mrl)
        Adds resource to the repository.
        Parameters:
        mrl - the resource to add