public interface Repository
Repository
is a format for storing data Artifact
s for various uses including deep
learning models and datasets.
This repository format is based off of the design of the Maven Repository format (See maven).
Unlike in Maven, the data doesn't need to be located within the repository. Instead, the
repository only stores metadata including the URL and checksum of the actual data. When the
artifact is prepared, the data is downloaded, checked, and then stored in the ~/.djo-ai/cache
folder.
The artifacts are first divided into a number of Metadata
files that can each have
multiple artifacts. The metadata files are identified by an MRL
which contains:
Application
).
Within each metadata are a number of artifacts that share the same groupId, artifactId, name, description, website, and update date. The artifacts within the metadata differ primarily based on name and properties. Note that there is a metadata name and a separate artifact name. The properties are a map with string property names and string property values that can be used to represent key differentiators between artifacts such as dataset, flavors, and image sizes. For example, you might have a ResNet metadata file with different artifacts to represent different hyperparameters and datasets used for training the ResNet.
Each artifact contains a Version
number (which can be a snapshot version). The data in
the artifacts are represented by files in the format of an Artifact.Item
and a parsed
JSON object of arguments. The files can either by a single file, an automatically extracted gzip
file, or an automatically extracted zip file that will be treated as a directory. These can be
used to store data such as the dataset, model parameters, and synset files. The arguments can be
used to store data about the model used for initialization. For example, it can store the image
size which can be used by the model loader for both initializing the block and setting up
resizing in the translator.
There are three kinds of repositories: a LocalRepository
, RemoteRepository
,
and SimpleRepository
. For all three kinds, new repositories should be created by calling
newInstance(String, String)
with the location of the repository.
Modifier and Type | Method and Description |
---|---|
java.net.URI |
getBaseUri()
Returns the URI to the base of the repository.
|
java.nio.file.Path |
getCacheDirectory()
Returns the cache directory for the repository.
|
java.nio.file.Path |
getFile(Artifact.Item item,
java.lang.String path)
Returns the path to a file for the item.
|
java.lang.String |
getName()
Returns the repository name.
|
default java.nio.file.Path |
getResourceDirectory(Artifact artifact)
Returns the resource directory for the an artifact.
|
java.util.List<MRL> |
getResources()
Returns a list of
MRL s in the repository. |
boolean |
isRemote()
Returns whether the repository is remote repository.
|
java.lang.String[] |
listDirectory(Artifact.Item item,
java.lang.String path)
Returns the list of files directly within a specified directory in a zipped directory item.
|
Metadata |
locate(MRL mrl)
Returns the metadata at a mrl.
|
static Repository |
newInstance(java.lang.String name,
java.nio.file.Path path)
Creates a new instance of a repository with a name and url.
|
static Repository |
newInstance(java.lang.String name,
java.lang.String url)
Creates a new instance of a repository with a name and url.
|
java.io.InputStream |
openStream(Artifact.Item item,
java.lang.String path)
Returns an
InputStream for an item in a repository. |
default void |
prepare(Artifact artifact)
Prepares the artifact for use.
|
void |
prepare(Artifact artifact,
ai.djl.util.Progress progress)
Prepares the artifact for use with progress tracking.
|
static void |
registerRepositoryFactory(RepositoryFactory factory)
Registers a
RepositoryFactory to handle the specified url scheme. |
Artifact |
resolve(MRL mrl,
java.lang.String version,
java.util.Map<java.lang.String,java.lang.String> filter)
Returns the artifact matching a mrl, version, and property filter.
|
static Repository newInstance(java.lang.String name, java.nio.file.Path path)
name
- the repository namepath
- the repository locationstatic Repository newInstance(java.lang.String name, java.lang.String url)
name
- the repository nameurl
- the repository locationstatic void registerRepositoryFactory(RepositoryFactory factory)
RepositoryFactory
to handle the specified url scheme.factory
- the RepositoryFactory
to be registeredboolean isRemote()
java.lang.String getName()
java.net.URI getBaseUri()
Metadata locate(MRL mrl) throws java.io.IOException
mrl
- the mrl of the metadata to retrievejava.io.IOException
- if it failed to load the metadataArtifact resolve(MRL mrl, java.lang.String version, java.util.Map<java.lang.String,java.lang.String> filter) throws java.io.IOException
mrl
- the mrl to match the artifact againstversion
- the version of the artifactfilter
- the property filterjava.io.IOException
- if it failed to load the artifactjava.io.InputStream openStream(Artifact.Item item, java.lang.String path) throws java.io.IOException
InputStream
for an item in a repository.item
- the item to openpath
- the path to a file if the item is a zipped directory. Otherwise, pass nulljava.io.IOException
- if it failed to open the streamjava.nio.file.Path getFile(Artifact.Item item, java.lang.String path) throws java.io.IOException
item
- the item to find the path forpath
- the path to a file if the item is a zipped directory. Otherwise, pass nulljava.io.IOException
- if it failed to find the pathjava.lang.String[] listDirectory(Artifact.Item item, java.lang.String path) throws java.io.IOException
item
- the zipped directory itempath
- the path within the zip directoryjava.io.IOException
- if it failed to list the directorydefault void prepare(Artifact artifact) throws java.io.IOException
artifact
- the artifact to preparejava.io.IOException
- if it failed to preparevoid prepare(Artifact artifact, ai.djl.util.Progress progress) throws java.io.IOException
artifact
- the artifact to prepareprogress
- the progress trackerjava.io.IOException
- if it failed to preparejava.nio.file.Path getCacheDirectory() throws java.io.IOException
java.io.IOException
- if it failed to ensure the creation of the cache directorydefault java.nio.file.Path getResourceDirectory(Artifact artifact) throws java.io.IOException
artifact
- the artifact whose resource directory to returnjava.io.IOException
- if it failed to ensure the creation of the cache directory