smile.classification

Class Maxent

java.lang.Object

smile.classification.Maxent

All Implemented Interfaces:

java.io.Serializable, java.util.function.ToDoubleFunction<int[]>, java.util.function.ToIntFunction<int[]>, Classifier<int[]>, OnlineClassifier<int[]>, SoftClassifier<int[]>

Direct Known Subclasses:

Maxent.Binomial, Maxent.Multinomial

public abstract class Maxent
extends java.lang.Object
implements SoftClassifier<int[]>, OnlineClassifier<int[]>

Maximum Entropy Classifier. Maximum entropy is a technique for learning probability distributions from data. In maximum entropy models, the observed data itself is assumed to be the testable information. Maximum entropy models don't assume anything about the probability distribution other than what have been observed and always choose the most uniform distribution subject to the observed constraints.

Basically, maximum entropy classifier is another name of multinomial logistic regression applied to categorical independent variables, which are converted to binary dummy variables. Maximum entropy models are widely used in natural language processing. Here, we provide an implementation which assumes that binary features are stored in a sparse array, of which entries are the indices of nonzero features.

See Also:

GLM,

References

1. A. L. Berger, S. D. Pietra, and V. J. D. Pietra. A maximum entropy approach to natural language processing. Computational Linguistics 22(1):39-71, 1996.

, Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Maxent.Binomial Binomial maximum entropy classifier.
static class	Maxent.Multinomial Multinomial maximum entropy classifier.

Constructor Summary

Constructors

Constructor and Description
Maxent(int p, double L, double lambda, smile.util.IntSet labels) Constructor.

Method Summary

Modifier and Type	Method and Description
double	AIC() Returns the AIC score.
static Maxent.Binomial	binomial(int p, int[][] x, int[] y) Learn maximum entropy classifier.
static Maxent.Binomial	binomial(int p, int[][] x, int[] y, double lambda, double tol, int maxIter) Learn maximum entropy classifier.
static Maxent.Binomial	binomial(int p, int[][] x, int[] y, java.util.Properties prop) Learn maximum entropy classifier.
int	dimension() Returns the dimension of input space.
static Maxent	fit(int p, int[][] x, int[] y) Learn maximum entropy classifier.
static Maxent	fit(int p, int[][] x, int[] y, double lambda, double tol, int maxIter) Learn maximum entropy classifier.
static Maxent	fit(int p, int[][] x, int[] y, java.util.Properties prop) Learn maximum entropy classifier.
double	getLearningRate() Returns the learning rate of stochastic gradient descent.
double	loglikelihood() Returns the log-likelihood of model.
static Maxent.Multinomial	multinomial(int p, int[][] x, int[] y) Learn maximum entropy classifier.
static Maxent.Multinomial	multinomial(int p, int[][] x, int[] y, double lambda, double tol, int maxIter) Learn maximum entropy classifier.
static Maxent.Multinomial	multinomial(int p, int[][] x, int[] y, java.util.Properties prop) Learn maximum entropy classifier.
void	setLearningRate(double rate) Sets the learning rate of stochastic gradient descent.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface smile.classification.SoftClassifier

predict

Methods inherited from interface smile.classification.OnlineClassifier

update, update

Methods inherited from interface smile.classification.Classifier

applyAsDouble, applyAsInt, f, predict, predict

Constructor Detail

Maxent

public Maxent(int p,
              double L,
              double lambda,
              smile.util.IntSet labels)

Constructor.

Parameters:

p - the dimension of input data.

L - the log-likelihood of learned model.

lambda - λ > 0 gives a "regularized" estimate of linear weights which often has superior generalization performance, especially when the dimensionality is high.

labels - class labels

Method Detail

fit

public static Maxent fit(int p,
                         int[][] x,
                         int[] y)

Learn maximum entropy classifier.

Parameters:

p - the dimension of feature space.

x - training samples. Each sample is represented by a set of sparse binary features. The features are stored in an integer array, of which are the indices of nonzero features.

y - training labels in [0, k), where k is the number of classes.

fit

public static Maxent fit(int p,
                         int[][] x,
                         int[] y,
                         java.util.Properties prop)

Learn maximum entropy classifier.

Parameters:

p - the dimension of feature space.

x - training samples. Each sample is represented by a set of sparse binary features. The features are stored in an integer array, of which are the indices of nonzero features.

y - training labels in [0, k), where k is the number of classes.

fit

public static Maxent fit(int p,
                         int[][] x,
                         int[] y,
                         double lambda,
                         double tol,
                         int maxIter)

Learn maximum entropy classifier.

Parameters:

p - the dimension of feature space.

x - training samples. Each sample is represented by a set of sparse binary features. The features are stored in an integer array, of which are the indices of nonzero features.

y - training labels in [0, k), where k is the number of classes.

lambda - λ > 0 gives a "regularized" estimate of linear weights which often has superior generalization performance, especially when the dimensionality is high.

tol - the tolerance for stopping iterations.

maxIter - maximum number of iterations.

binomial

public static Maxent.Binomial binomial(int p,
                                       int[][] x,
                                       int[] y)

Learn maximum entropy classifier.

Parameters:

p - the dimension of feature space.

x - training samples. Each sample is represented by a set of sparse binary features. The features are stored in an integer array, of which are the indices of nonzero features.

y - training labels in [0, k), where k is the number of classes.

binomial

public static Maxent.Binomial binomial(int p,
                                       int[][] x,
                                       int[] y,
                                       java.util.Properties prop)

Learn maximum entropy classifier.

Parameters:

p - the dimension of feature space.

x - training samples. Each sample is represented by a set of sparse binary features. The features are stored in an integer array, of which are the indices of nonzero features.

y - training labels in [0, k), where k is the number of classes.

binomial

public static Maxent.Binomial binomial(int p,
                                       int[][] x,
                                       int[] y,
                                       double lambda,
                                       double tol,
                                       int maxIter)

Learn maximum entropy classifier.

Parameters:

p - the dimension of feature space.

x - training samples. Each sample is represented by a set of sparse binary features. The features are stored in an integer array, of which are the indices of nonzero features.

y - training labels in [0, k), where k is the number of classes.

lambda - λ > 0 gives a "regularized" estimate of linear weights which often has superior generalization performance, especially when the dimensionality is high.

tol - the tolerance for stopping iterations.

maxIter - maximum number of iterations.

multinomial

public static Maxent.Multinomial multinomial(int p,
                                             int[][] x,
                                             int[] y)

Learn maximum entropy classifier.

Parameters:

p - the dimension of feature space.

x - training samples. Each sample is represented by a set of sparse binary features. The features are stored in an integer array, of which are the indices of nonzero features.

y - training labels in [0, k), where k is the number of classes.

multinomial

public static Maxent.Multinomial multinomial(int p,
                                             int[][] x,
                                             int[] y,
                                             java.util.Properties prop)

Learn maximum entropy classifier.

Parameters:

p - the dimension of feature space.

x - training samples. Each sample is represented by a set of sparse binary features. The features are stored in an integer array, of which are the indices of nonzero features.

y - training labels in [0, k), where k is the number of classes.

multinomial

public static Maxent.Multinomial multinomial(int p,
                                             int[][] x,
                                             int[] y,
                                             double lambda,
                                             double tol,
                                             int maxIter)

Learn maximum entropy classifier.

Parameters:

p - the dimension of feature space.

x - training samples. Each sample is represented by a set of sparse binary features. The features are stored in an integer array, of which are the indices of nonzero features.

y - training labels in [0, k), where k is the number of classes.

lambda - λ > 0 gives a "regularized" estimate of linear weights which often has superior generalization performance, especially when the dimensionality is high.

tol - the tolerance for stopping iterations.

maxIter - maximum number of iterations.

dimension

public int dimension()

Returns the dimension of input space.

Returns:

the dimension of input space.

setLearningRate

public void setLearningRate(double rate)

Sets the learning rate of stochastic gradient descent. It is a good practice to adapt the learning rate for different data sizes. For example, it is typical to set the learning rate to eta/n, where eta is in [0.1, 0.3] and n is the size of the training data.

Parameters:

rate - the learning rate.

getLearningRate

public double getLearningRate()

Returns the learning rate of stochastic gradient descent.

loglikelihood

public double loglikelihood()

Returns the log-likelihood of model.

AIC

public double AIC()

Returns the AIC score.