All Classes and Interfaces
Class
Description
This class is the entry point for processing operations.
Mock durations for different Actions.
Performs the Action and maintenance around the Action such as
- Setting the start time and saving the operation's duration
- Executing pauses before and after the operation
- Determining the success of the operation
- Saving the MatchSnapshots
Any Action that is run independently of this class should
have code to perform maintenance around the Action.
Actions that are run from the Action class need to follow this interface.
ActionOptions provides options for configuring an action.
BasicActions:
FIND
CLICK
DEFINE return a Region with specific x,y,w,h
TYPE sends keyboard input
MOVE moves the mouse
VANISH is successful when an Image or State disappears
GET_TEXT reads text from a Region
HIGHLIGHT highlights a Match, Region, or Location
SCROLL_MOUSE_WHEEL
MOUSE_DOWN
MOUSE_UP
KEY_DOWN
KEY_UP
CLASSIFY
CompositeActions:
CLICK_UNTIL clicks Matches, Regions, and/or Locations until a condition is fulfilled
DRAG
Specifies the condition to fulfill after a Click.
KMEANS finds a selected number of RGB color cluster centers for each image.
Anchors define Locations in Matches and specify how these Locations should be used
to define a Region (see the Anchor class for more info).
Images can contain multiple Patterns.
Keep in mind:
ObjectCollections can contain multiple Images.
TEXT_APPEARS: Keep searching for text until some text appears.
For scrolling with the mouse wheel
An ActionResultsCombos has both an action and an expected result.
Returns the corresponding Action class as specified in ActionOptions.
Non-image objects are adjusted and added directly to the Matches variable.
Snapshots are created only when an appropriate Attribute is active on the page
(i.e.
Adjust Matches according to the options in ActionOptions.
After a successful click,
Sets the probability that the object remains visible or disappears.
This class is a central location for pixel analysis, for every color-based match strategy.
Anchors define
1) A Position in the Image Match or Region
2) A Position in the Region to define
A point defined with Position #1 is used to define part of a Region.
Gets the window of the App in focus.
AttributeData keeps track of the following info:
- the Attribute is active on which pages
- the results of searching for the Image on each page with respect to this Attribute
Here, the term 'page' refers to screenshots that are used to create the initial
State structure.
Repository of BabyStates.
BasicActions, which run for 1 iteration, require 1 or no Find operations.
Global settings
Some Brobot settings override Sikuli settings.
Builds a StateStructure using labeled images and screenshots of the automation environment.
The scene is captured along with the time of the capture.
Saves a series of screenshots.
DynamicImages classify as probabilities, standard Images have 100% pixel probability when found.
Clicks on an Image Match, Region, or Location.
Wrapper class for Click, handles real and mock clicks.
Converts ClickType to a Sikuli Button.
ClickUntil clicks Matches until a condition occurs or the operation times out.
When working with color profiles, images should be created that represent the colors
to be found.
ColorInfo has the min, max, mean, and stddev for a single ColorValue, such as Hue, Saturation, or Value.
A ColorSchema is a representation of a color cluster.
ColorStatProfile is a color in a specific color space (HSV, BGR, etc.).
It is useful to create one or more CommonActions classes for your applications
in order to reduce code redundancy and make your code more readable.
Currently ActionResultCombos are set up for click operations as the first action.
This class defines some typical configurations for Drag and
allows for multiple drag operations.
This is an example of a more complex custom action.
This class should make it simpler to create common StateTransitions.
CompositeActions are built from BasicActions.
Performs nested Find actions, in the order they appear in the FindActions list in
ActionOptions.
This class creates a deep copy of most of the fields in ActionOptions.
DanglingSnapshots keep Snapshots with their objects while the operation is in progress.
Each region has 4 borders that can be defined: top, bottom, left, right.
Helper functions for various Define classes
Defines a Region as the largest rectangle including all Matches and Locations.
Defines a Region as the smallest rectangle produced by Matches and Locations.
Defines a Region as the largest rectangle produced by Matches and Locations.
Defines a Region with the Define method specified in the ActionOptions.
Defines a Region around a Match.
Defines a Region using the active Window as a reference.
Completes a whole Transition (FromTransition and ToTransition),
including all States to activate and exit.
The DoUntilActionObject contains both ActionOptions and ObjectCollections for both
'do' and 'until' Actions.
Drags from an Image Match, Region, or Location to an Image Match, Region, or Location
Drag is a Composite Action composed of a Basic Action (Find) and a Sikuli Wrapper (Drag Location)
Wrapper class for Drag, handles real and mock drags.
A DynamicImage is an image that does not have a fixed pattern, and thus cannot be accurately found
with SikuliX or OpenCV pattern matching.
Defines the condition necessary to stop repeating sequences for individual Actions.
IllustrationScenes store their filenames before they are written to file.
All find requests come here first and are then sent to a specific type of find method.
Finds all Matches for all Patterns in the Brobot Image.
The steps for color classification are:
1.
Returns Matches for the first Pattern found in a Brobot Image.
Retrieve a Find function with its Find type.
Implements the Find.FIRST and Find.ALL methods for Images.
Sends the Image object to either FindImage or FindRIP depending on whether the Image location can vary.
Works with all Images in a State for a specific screenshot/page
- Finds all occurrences of an image in a screenshot
- Calls the methods necessary for processing Attributes and adding Snapshots
- Finishes the GroupDefine operation
TimeBetweenIndividualActions is the option in ActionOptions that gives the time (in seconds)
between the first, second, and third screenshots.
Contains functions to find one or all Matches given an Image and a Region.
RegionImagePairs are pairs of (Region, Image)
The Region in a RegionImagePairs is defined once an Image is found.
Returns the class corresponding to the selected Find option.
For determining active Attributes for an Image on a specific page.
Returns the 'from' and 'to' Locations to be used in Drag.
Retrieves files as Strings (in the case of screenshots) or
as StateImageObjects (in the case of images).
The same methods as in GetImage, but for JavaCV.
Scenes are used for
- finding color
- image segmentation (classification)
- action illustration
When running live, scenes should be taken in every Find iteration.
GetText retrieves text from a Region.
Wrapper class for GetText, handles real and mock text queries.
Grid is either defined by columns and row, or by cell width and cell height.
Highlight all Matches at once, or one at a time.
Wrapper class for Highlight, performs real or mock highlights.
Has the mask for one part of each image, i.e.
Divides a region into five subregions to facilitate histogram and color Find operations.
Images can hold multiple Patterns.
Attributes learned from the image filename and initial screenshots of the environment.
Writes Java code for a StateImageObject
Collects Images and Match objects from Images with an active GROUP_DEFINE Attribute.
Helper functions for working with Images.
Functions for working with Image Patterns
Given sets of possible active States, this class searches for these States
to set active States.
Uses only ObjectCollection #1
An ObjectCollection can have multiple keys
ActionObjects hold special keys such as CTRL that are pressed first
KeyDown in SikuliX (Java Robot) does not actually hold the key down.
Uses only ObjectCollection #1
An ObjectCollection can have multiple keys
ActionObjects hold special keys such as CTRL that are released last
Wrapper class for KeyUp, works for real or mocked operations.
KmeansCluster represents a cluster of points in a k-means analysis corresponding to one of the centers.
KmeansProfile represents a k-means analysis for one number of means and one color schema (BGR, HSV, etc.).
Creates a series of kmeans profiles for commonly used number of centers (2-10)
Calculating this up-front will save time and CPU/GPU.
Location can be an absolute position (x,y) on the screen,
or a relative position (%w, %h) of a Region.
Writes Java code for a StateLocation.
This builder puts a bunch of Mats together.
The results object for all actions.
Used for mock runs when not empty; otherwise, the State's probability variable is used.
Creates Match objects when probability is being used instead of Snapshots.
MatchObject is used to store information about new Image matches.
MatchSnapshots record a match (or failed match) and the search options at a single point in time.
3d versions of common OpenCV operations that only accept one channel.
Mainly for testing, this class provides code to easily visualize Mats.
The Action methods should be unaware of the mock process, which should occur only
at the interface between Brobot and Sikuli, where the Wrapper classes live.
Mock Matches can be found with 1 of 2 methods:
History: Using the MatchSnapshots associated with individual Images.
Takes care of setting probabilities for States and State objects
Keeps track of the number of mocks performed.
Mock text for GetText Actions using the probability method (and not Snapshots).
Presses and holds a mouse button.
Wrapper class for MouseDown, works for real or mock actions.
Releases a mouse button.
Wrapper class for MouseUp, works for real or mocked actions.
Wrapper class for MouseWheel, works for real or mocked actions.
Moves the mouse to one or more locations.
Wrapper class for MoveMouse, works for real or mocked actions.
Helper function for Composite Actions
NOT WORKING! Sikuli appears not to allow mouse movement after a mouse down operation.
This is an example of how to create an Action using the MultipleActions class.
A demonstration of how to use the JNativeHook library.
Performs nested Find actions, in the order they appear in the FindActions list in
ActionOptions.
NullState is used as a generic State for passing simple objects
(non State objects) to Actions.
This class holds all the objects that can be passed to an Action.
ParameterCollections help Brobot calibrate specific parameters with respect
to success conditions.
A single path from a start State to a target State.
Finds all paths from a set of start States to a target State.
If a Path is unsuccessfully traversed, we find ourselves somewhere in the middle of
the Path.
A list of Path objects comprising the possible ways to go from
start States to the target State.
Holds the results of all per-pixel color analysis for a {scene, ColorProfile} pair.
Contains a series of PixelAnalysis objects that comprise all analysis of a {scene, StateImageObject} pair.
Builds a State structure by comparing the images found per screenshot with the existing
State structure at that point.
Position is defined by (%w, %h) and can be used to get a location in a Region.
Prints Attribute stats during the creation of the State structure.
It divides a region into
grid cells, some overlapping, and counts the points that are within the cells.
Region extends the Sikuli class Region and adds, among other functionality, new
initializers, analysis tools, and points x2 and y2.
Writes Java code for a StateRegion.
An Image with an associated Region.
RegionImagePairs contain Images that should appear in fixed locations.
Analysis functions involving Regions
Prints to the console if the output meets the required reporting level.
Returns a List with the first element the Action Matches and the second element the Results Matches
ActionResultsCombos are typically:
- an Action such as CLICK
- a Result from an operation such as FIND
The parameters and results only get recorded if the Action is successful.
A collection of PixelAnalysisCollection objects, each of which contains all analysis of a {scene, StateImageObject} pair.
Scrolls the mouse wheel up or down.
SearchRegions allows for multiple Regions to be associated with an Image.
Select is a custom Action built using the custom SelectActionObject.
Determines the State and Image names, Transition targets, and Attributes from the filename.
Set the hidden States after a successful State transition.
Mats used to illustrate the image and its color profiles can be stored together with
the image and referenced for illustration when needed.
Writes Java code for MatchSnapshots for a single StateImageObject.
States give a Brobot application structure.
Writes Java code for the declaration of a new State (everything after 'State state =').
Writes Java code for the remaining components of a State class:
- the State variable (State state = new State.Builder(ENUM)...)
- the initialization of this variable 'stateService.save(state)'.
This class finds the active States after Brobot is lost.
StateImageObject represents an Image that exists in the owner State.
A StateLocation belongs to a State and usually has a Location that
has a special meaning for its owner State.
StateMemory keeps track of which States are currently active.
The MatchHistory keep a record of the StateObject's Snapshots.
A StateRegion belongs to a State and usually has a Region that
has a special meaning for its owner State.
Manages the State repository.
A State String belongs to a State and usually has a String that
has a special meaning for the owner State.
StateText is text that always appears in its owner State.
After a successful transition (both 'from' and 'to' Transitions):
'activate' holds all States to make active
'exit' holds all States to deactivate
StateTransitions hold all the Transitions for a State.
Holds all static incoming and outgoing Transitions,
plus the variable transitions to hidden States.
We want to go to a specific State, the target State.
Manages the StateTransitions repository and retrieves
StateTransitions given a StateEnum.
Finds the correct Transition from one State to another,
taking into account the hidden States.
Calculates the similarity of 2 Strings.
Defines the success condition for individual Actions.
Tests can be used for mocks or for live execution.
Text read from the screen is a stochastic variable.
The TextSelector returns text from the stochastic Text variable (Strings are stored in a List),
which it believes has the highest probability of being the actual text on the screen.
Deals with Action durations, both real and mock.
Wraps the Time functions to allow for mocking actions.
Writes variable declarations for classes referenced with dependency injection.
This class returns an object containing the StateTransitions objects and StateTransition objects that
we are interested in for a specific transition from one State to another.
Writes Java code for methods in the StateTransitions class.
Writes Java code for the StateTransitions class, including static imports
for all enums corresponding to the target States of Transitions.
Moves through the Paths to reach the target State.
Aims to visualize the State hierarchy.
Types text to the window in focus.
Wrapper class for TypeText, works with real or mocked actions.
The Unknown State is usually the start point for any application,
and can also be found when Brobot has become lost.
Processes every active Attribute for an Image on a page after a Find operation.
Use the defined regions (if they exist) of objects as MatchObjects
Wrapper class for Wait, works with real or mocked actions.
Returns a successful Matches object if at some point no objects are found.
Creates the folder 'stateStructure'.
Prepares all names (class, package, enum, variable, etc) and calls the methods that
write the Java code for both State and StateTransitions classes.
Writes Java code for the State class.
Writes Java code for the StateTransitions class.