Package org.evomaster.client.java.controller.internal

Class SutController

java.lang.Object
org.evomaster.client.java.controller.internal.SutController
All Implemented Interfaces:
CustomizationHandler, SutHandler
Direct Known Subclasses:
EmbeddedSutController, ExternalSutController
public abstract class SutController extends Object implements SutHandler, CustomizationHandler
Abstract class used to connect to the EvoMaster process, and that is responsible to start/stop/restart the tested application, ie the system under test (SUT)

  • Constructor Details

    • SutController

      public SutController()

  • Method Details

    • startTheControllerServer

      public final boolean startTheControllerServer()
      Start the controller as a RESTful server. Use the setters of this class to change the default port and host.
      This method is blocking until the server is initialized.
      Returns:
      true if there was no problem in starting the controller

    • stopTheControllerServer

      public final boolean stopTheControllerServer()

    • getControllerServerPort

      public final int getControllerServerPort()
      Returns:
      the actual port in use (eg, if it was an ephemeral 0)

    • getControllerPort

      public final int getControllerPort()

    • setControllerPort

      public final void setControllerPort(int controllerPort)

    • getControllerHost

      public final String getControllerHost()

    • setControllerHost

      public final void setControllerHost(String controllerHost)

    • deriveObjectParameterData

      public String deriveObjectParameterData(String paramName, String jsonObject, String endpointPath) throws Exception
      Function used to derived params from the other values in the same object.
      Parameters:
      paramName - the name of the parameter we need to derive
      jsonObject - JSON representation of full object evolved in EM. You might need just a subset to derive needed param
      endpointPath - the path of endpoint for this object, in case need to distinguish between same params in different endpoints
      Returns:
      a string representation of derived value for paramName
      Throws:
      Exception

    • execInsertionsIntoDatabase

      public InsertionResultsDto execInsertionsIntoDatabase(List<InsertionDto> insertions, InsertionResultsDto... previous)
      Description copied from interface: SutHandler
      Execute the given data insertions into the database (if any)
      Specified by:
      execInsertionsIntoDatabase in interface SutHandler
      Parameters:
      insertions - DTOs for each insertion to execute
      previous - an array of insertion results which were executed before this execution
      Returns:
      insertion execution results

    • execInsertionsIntoMongoDatabase

      public MongoInsertionResultsDto execInsertionsIntoMongoDatabase(List<MongoInsertionDto> insertions)
      Specified by:
      execInsertionsIntoMongoDatabase in interface SutHandler

    • getActionIndex

      public int getActionIndex()

    • enableComputeSqlHeuristicsOrExtractExecution

      public final void enableComputeSqlHeuristicsOrExtractExecution(boolean enableSqlHeuristics, boolean enableSqlExecution, boolean advancedHeuristics)

    • initSqlHandler

      public final void initSqlHandler()
      This is needed only during test generation (not execution), and it is automatically called by the EM controller after the SUT is started.

    • initMongoHandler

      public final void initMongoHandler()

    • initOpenSearchHandler

      public final void initOpenSearchHandler()

    • getConnectionIfExist

      public final Connection getConnectionIfExist()
      TODO further handle multiple connections
      Returns:
      sql connection if there exists

    • doEmploySmartDbClean

      public final boolean doEmploySmartDbClean()
      Returns:
      whether to employ smart db clean

    • resetExtraHeuristics

      public final void resetExtraHeuristics()

    • getExtraHeuristics

      public final List<ExtraHeuristicsDto> getExtraHeuristics(boolean queryFromDatabase)
      Parameters:
      queryFromDatabase - specifies whether to compute extra heuristics by retrieving data from database. if true, computing such heuristics will query all data in the related databases.
      Returns:
      a list of dto representing computed extra heuristic

    • computeExtraHeuristics

      public final ExtraHeuristicsDto computeExtraHeuristics(boolean queryFromDatabase)

    • computeMongoHeuristics

      public final void computeMongoHeuristics(ExtraHeuristicsDto dto, List<AdditionalInfo> additionalInfoList)

    • computeOpenSearchHeuristics

      public final void computeOpenSearchHeuristics(ExtraHeuristicsDto dto, List<AdditionalInfo> additionalInfoList)

    • registerOrExecuteInitSqlCommandsIfNeeded

      public final void registerOrExecuteInitSqlCommandsIfNeeded()
      handle specified init sql script after SUT is started.
      Specified by:
      registerOrExecuteInitSqlCommandsIfNeeded in interface SutHandler

    • registerOrExecuteInitSqlCommandsIfNeeded

      public final void registerOrExecuteInitSqlCommandsIfNeeded(boolean instrumentationEnabled)
      handle specified init sql script after SUT is started.
      Parameters:
      instrumentationEnabled - represents whether the instrumentation is enabled

    • cleanAccessedTables

      public final void cleanAccessedTables()
      perform smart db clean by cleaning the data in accessed table

    • addTableToInserted

      public void addTableToInserted(List<String> tables)
      collect info about what table are manipulated by evo in order to generate data directly into it
      Parameters:
      tables - a list of name of tables

    • addSuccessfulInitSqlInsertion

      public void addSuccessfulInitSqlInsertion(InsertionDto insertionDto)
      collect info about init sql insertion which has been executed succesfully
      Parameters:
      insertionDto - a dto for sql insertion

    • getSqlDatabaseSchema

      public final org.evomaster.client.java.controller.api.dto.database.schema.DbInfoDto getSqlDatabaseSchema()
      Extra information about the SQL Database Schema, if any is present. Note: this is extracted by querying the database itself. So the database must be up and running.
      Returns:
      a DTO with the schema information
      See Also:

    • extractSqlDbSchemaAndConstraints

      public final boolean extractSqlDbSchemaAndConstraints()
      Returns:
      if the extraction of SQL schema and constraints with the given DbSpecification has been proceeded successfully Note that such extraction is only performed once. if the extraction has been performed, it returns false.

    • getRPCSchema

      public final Map<String,InterfaceSchema> getRPCSchema()
      Returns:
      a map from the name of interface to extracted interface

    • getLocalAuthSetupSchemaMap

      public Map<Integer,LocalAuthSetupSchema> getLocalAuthSetupSchemaMap()
      Returns:
      a map of auth local method

    • extractRPCProblemDto

      public RPCProblemDto extractRPCProblemDto(boolean isSutRunning)

    • extractRPCSchema

      public final void extractRPCSchema()
      extract endpoints info of the RPC interface by reflection based on the specified service interface name
      Specified by:
      extractRPCSchema in interface SutHandler

    • handleSeededTests

      public Map<String,RPCTestDto> handleSeededTests(boolean isSUTRunning)
      parse seeded tests for RPC
      Returns:
      seeded tests with a map, key is a name of the seeded test case, value is a list of RCPActionDto for the test case

    • verifySqlConnection

      @Deprecated public final boolean verifySqlConnection()
      Deprecated.
      Either there is no connection, or, if there is, then it must have P6Spy configured. But this might not apply to all kind controllers
      Returns:
      false if the verification failed

    • newSearch

      public abstract void newSearch()
      Re-initialize all internal data to enable a completely new search phase which should be independent from previous ones

    • postSearchAction

      public void postSearchAction(PostSearchActionDto dto)
      handling post actions after the search
      Parameters:
      dto - contains required info for the post handling

    • newTest

      public final void newTest()
      Re-initialize some internal data needed before running a new test

    • newAction

      public final void newAction(ActionDto dto, boolean queryFromDatabase)
      As some heuristics are based on which action (eg HTTP call, or click of button) in the test sequence is executed, and their order, we need to keep track of which action does cover what.
      Parameters:
      dto - the DTO with the information about the action (eg its index in the test)

    • newScheduleAction

      public final void newScheduleAction(ScheduleTaskInvocationDto dto, boolean queryFromDatabase)

    • executeHandleLocalAuthenticationSetup

      public final void executeHandleLocalAuthenticationSetup(RPCActionDto dto, ActionResponseDto responseDto)

    • invokeScheduleTasks

      public final void invokeScheduleTasks(List<ScheduleTaskInvocationDto> dtos, ScheduleTaskInvocationsResult responseDto, boolean queryFromDatabase)

    • executeAction

      public final void executeAction(RPCActionDto dto, ActionResponseDto responseDto)
      execute a RPC request based on the specified dto
      Parameters:
      dto - is the action DTO to be executed

    • executeRPCEndpoint

      public Object executeRPCEndpoint(String json) throws Exception
      Description copied from interface: SutHandler

      execute an RPC endpoint with evomaster driver

      Specified by:
      executeRPCEndpoint in interface SutHandler
      Parameters:
      json - contains info of an RPC endpoint
      Returns:
      value returned by this execution. it is nullable.
      Throws:
      Exception

    • newTestSpecificHandler

      public abstract void newTestSpecificHandler()

    • newActionSpecificHandler

      public abstract void newActionSpecificHandler(ActionDto dto)

    • newScheduleActionSpecificHandler

      public abstract void newScheduleActionSpecificHandler(ScheduleTaskInvocationDto dto)

    • isInstrumentationActivated

      public abstract boolean isInstrumentationActivated()
      Check if bytecode instrumentation is on.
      Returns:
      true if the instrumentation is on

    • isSutRunning

      public abstract boolean isSutRunning()

      Check if the system under test (SUT) is running and fully initialized

      How to implement this method depends on the library/framework used to build the application. In Spring applications, this can be done with something like: ctx != null && ctx.isRunning(), where ctx is a field where ConfigurableApplicationContext should be stored when starting the application.

      Returns:
      true if the SUT is running

    • getPackagePrefixesToCover

      public abstract String getPackagePrefixesToCover()

      A "," separated list of package prefixes or class names. For example, "com.foo.,com.bar.Bar". This is used to specify for which classes we want to measure code coverage.

      Note: be careful of using something as general as "com." or "org.", as most likely ALL your third-party libraries would be instrumented as well, which could have a severe impact on performance.

      Returns:
      a String representing the packages to cover

    • getInfoForAuthentication

      public abstract List<AuthenticationDto> getInfoForAuthentication()

      If the application uses some sort of authentication, these details need to be provided here. Even if EvoMaster can have access to the database, it would not be able to recover hashed passwords.

      To test the application, there is the need to provide auth for at least 1 user (and more if they have different authorization roles). When EvoMaster generates test cases, it can decide to use the credential of any user provided by this method.

      What type of info to provide here depends on the auth mechanism, e.g., Basic or cookie-based (using LoginEndpoint). To simplify the creation of these DTOs with auth info, you can look at AuthUtils.

      If the credential are stored in a database, be careful on how the method resetStateOfSUT is implemented. If you delete all data with DbCleaner, then you will need as well to recreate the auth details. This can be put in a script, executed then with SqlScriptRunner.

      Returns:
      a list of valid authentication credentials, or null if * none is necessary

    • getConnection

      @Deprecated public final Connection getConnection()
      Deprecated.
      this is now set in DbSpecification

      If the system under test (SUT) uses a SQL database, we need to have a configured connection to access it.

      This method is related to SutHandler.resetStateOfSUT(). When accessing a Connection object to reset the state of the application, we suggest to save it to field (eg when starting the application), and return such field here, e.g., return connection;. This connection object will be used by EvoMaster to analyze the state of the database to create better test cases.

      Returns:
      null if the SUT does not use any SQL database

    • getDatabaseDriverName

      @Deprecated public final String getDatabaseDriverName()
      Deprecated.
      this method is no longer needed
      If the system under test (SUT) uses a SQL database, we need to specify the driver used to connect, eg. org.h2.Driver. This is needed for when we intercept SQL commands with P6Spy
      Returns:
      null if the SUT does not use any SQL database

    • getTargetInfos

      public abstract List<TargetInfo> getTargetInfos(Collection<Integer> ids, boolean fullyCovered, boolean descriptiveIds)

    • getAdditionalInfoList

      public abstract List<AdditionalInfo> getAdditionalInfoList()
      Returns:
      additional info for each action in the test. The list is ordered based on the action index.

    • getProblemInfo

      public abstract ProblemInfo getProblemInfo()

      Depending of which kind of SUT we are dealing with (eg, REST, GraphQL or SPA frontend), there is different info that must be provided. For example, in a RESTful API, you need to speficy where the OpenAPI/Swagger schema is located.

      The interface ProblemInfo provides different implementations, like RestProblem. You will need to instantiate one of such classes, and return it here in this method.

      Returns:
      an instance of object with all the needed data for the specific addressed problem

    • getPreferredOutputFormat

      public abstract SutInfoDto.OutputFormat getPreferredOutputFormat()
      Test cases could be outputted in different language (e.g., Java and Kotlin), using different testing libraries (e.g., JUnit 4 or 5). Here, need to specify the default option.
      Returns:
      the format in which the test cases should be generated

    • getUnitsInfoDto

      public abstract UnitsInfoDto getUnitsInfoDto()

    • setKillSwitch

      public abstract void setKillSwitch(boolean b)

    • setExecutingInitSql

      public abstract void setExecutingInitSql(boolean executingInitSql)

    • setExecutingInitMongo

      public abstract void setExecutingInitMongo(boolean executingInitMongo)

    • setExecutingAction

      public abstract void setExecutingAction(boolean executingAction)

    • bootingSut

      public abstract void bootingSut(boolean isBooting)
      specify whether the SUT is booting (ie starting up), or not. this is needed because we don't want to handle targets covered at startup during the fitness evaluations
      Parameters:
      isBooting -

    • getBootTimeInfoDto

      public abstract BootTimeInfoDto getBootTimeInfoDto()

    • getBootTimeInfoDto

      protected BootTimeInfoDto getBootTimeInfoDto(BootTimeObjectiveInfo info)

    • getJvmDtoSchema

      public abstract void getJvmDtoSchema(List<String> dtoNames)

    • getExecutableFullPath

      public abstract String getExecutableFullPath()

    • getUnitsInfoDto

      protected UnitsInfoDto getUnitsInfoDto(UnitsInfoRecorder recorder)

    • getRPCClient

      public Object getRPCClient(String interfaceName)
      Description copied from interface: SutHandler

      return an instance of a client of an RPC service.

      This method must be blocking until the SUT is initialized.

      This method is only required when the problem is RPC for the moment, otherwise return null

      might change string interfaceName to class interface
      Specified by:
      getRPCClient in interface SutHandler
      Parameters:
      interfaceName - a full name of an interface
      Returns:
      a client which could send requests to the interface

    • categorizeBasedOnResponse

      public CustomizedCallResultCode categorizeBasedOnResponse(Object response)
      Description copied from interface: CustomizationHandler

      categorize result based on response

      Specified by:
      categorizeBasedOnResponse in interface CustomizationHandler
      Parameters:
      response - is a given response
      Returns:
      a call result code based on the response

    • getCustomizedValueInRequests

      public List<CustomizedRequestValueDto> getCustomizedValueInRequests()
      Description copied from interface: CustomizationHandler

      specify candidate values in requests

      Specified by:
      getCustomizedValueInRequests in interface CustomizationHandler
      Returns:
      a list of specified values for requests

    • specifyCustomizedNotNullAnnotation

      public List<CustomizedNotNullAnnotationForRPCDto> specifyCustomizedNotNullAnnotation()
      Description copied from interface: CustomizationHandler

      specify customized annotation indicating that field in DTO is not null if any

      Specified by:
      specifyCustomizedNotNullAnnotation in interface CustomizationHandler
      Returns:
      a list of such annotation

    • seedRPCTests

      public List<SeededRPCTestDto> seedRPCTests()
      Description copied from interface: CustomizationHandler

      specify seeded tests for RPC

      Specified by:
      seedRPCTests in interface CustomizationHandler
      Returns:
      a list of dto for seeded tests

    • customizeRPCTestOutput

      public boolean customizeRPCTestOutput(RPCTestWithResultsDto rpcTest)
      Description copied from interface: CustomizationHandler

      here we provide additional handling on the generated RPC tests

      Specified by:
      customizeRPCTestOutput in interface CustomizationHandler
      Parameters:
      rpcTest - represent generated tests by evomaster which contains executed actions and executed results, ie, return, for deriving assertions
      Returns:
      a result of handling of additional RPC Test

    • customizeMockingRPCExternalService

      public boolean customizeMockingRPCExternalService(List<MockRPCExternalServiceDto> externalServiceDtos, boolean enabled)
      Description copied from interface: CustomizationHandler

      implement how to enable/disable mocking of RPC based external services

      Specified by:
      customizeMockingRPCExternalService in interface CustomizationHandler
      Parameters:
      externalServiceDtos - contains info about how to setup responses
      enabled - reflect to enable (set it true) or disable (set it false) the specified external service dtos. Note that null [externalServiceDtos] with false [enabled] means that all existing external service setup should be disabled.
      Returns:
      whether the mocked instance starts successfully,

    • customizeMockingDatabase

      public boolean customizeMockingDatabase(List<MockDatabaseDto> databaseDtos, boolean enabled)
      Description copied from interface: CustomizationHandler

      implement how to enable/disable customized mock objects for database

      Specified by:
      customizeMockingDatabase in interface CustomizationHandler
      Parameters:
      databaseDtos - contains info about how to mock databases based on commandName
      enabled - reflect to enable (set it true) or disable (set it false) mock objects for sql command based on commandName Note that null [databaseDtos] with false [enabled] means that all existing mock objects should be disabled.
      Returns:
      whether the mocked instance starts successfully,

    • customizeScheduleTaskInvocation

      public ScheduleTaskInvocationResultDto customizeScheduleTaskInvocation(ScheduleTaskInvocationDto invocationDto, boolean invoked)
      Description copied from interface: CustomizationHandler

      implement how to invoke schedule task for providing a customized solution regarding ScheduleTaskInvocationResultDto, its status typed with ExecutionStatusDto indicates invocation status of the schedule tasks Once the task is successfully invoked, its status should be RUNNING then if the task is completed, its status should be COMPLETE FAILED is used to handle any exception in handling invocation of the schedule task.

      Specified by:
      customizeScheduleTaskInvocation in interface CustomizationHandler
      Parameters:
      invocationDto - specified necessary info for invoking/terminating schedule tasks
      invoked - defines to invoke (invoked is true) or terminate (invoked is false) the specified schedule task
      Returns:
      invocation result dto

    • isScheduleTaskCompleted

      public boolean isScheduleTaskCompleted(ScheduleTaskInvocationResultDto invocationInfo)
      Specified by:
      isScheduleTaskCompleted in interface CustomizationHandler
      Parameters:
      invocationInfo - has the info about the invoked schedule task
      Returns:
      whether the task is completed

    • resetDatabase

      public void resetDatabase(List<String> tablesToClean)
      Description copied from interface: SutHandler

      reset database if the smart db cleaning is employed

      Specified by:
      resetDatabase in interface SutHandler
      Parameters:
      tablesToClean - represents a list of table which will be reset based on specified DbSpecification. note that null tablesToClean means all table will be reset.

    • packagesToSkipInstrumentation

      public String packagesToSkipInstrumentation()
      Comma , separated list of package prefixes of classes to skip. This is mainly used as workaround for cases in which EM's instrumentation crashes due to some bugs in it. (This is also the reason why it is not abstract) Note: we currently cannot test this in a E2E, as agent is loaded _before_ te controller is defined

    • resetCustomizedMethodForMockObject

      public final boolean resetCustomizedMethodForMockObject()

      a method to reset mocked external services with customized method

      Specified by:
      resetCustomizedMethodForMockObject in interface SutHandler

    • mockRPCExternalServicesWithCustomizedHandling

      public final boolean mockRPCExternalServicesWithCustomizedHandling(String externalServiceDtos, boolean enabled)

      a method to employ customized mocking of RPC based external services

      Specified by:
      mockRPCExternalServicesWithCustomizedHandling in interface SutHandler
      Parameters:
      externalServiceDtos - contains info about how to setup responses with json format, note that the json should be able to be converted to a list of MockRPCExternalServiceDto
      enabled - reflect to enable (set it true) or disable (set it false) the specified external service dtos. Note that null [externalServiceDtos] with false [enabled] means that all existing external service setup should be disabled.
      Returns:
      whether the mocked instance starts successfully,

    • mockDatabasesWithCustomizedHandling

      public boolean mockDatabasesWithCustomizedHandling(String mockDatabaseObjectDtos, boolean enabled)
      Description copied from interface: SutHandler

      a method to employ customized mocking for database

      Specified by:
      mockDatabasesWithCustomizedHandling in interface SutHandler
      Parameters:
      mockDatabaseObjectDtos - contains info about how to set up mock object for databases with json format, note that the json should be able to be converted to a list of MockDatabaseDto
      enabled - reflect to enable (set it true) or disable (set it false) the specified mock object Note that null [mockDatabaseObjectDtos] with false [enabled] means that all existing mock objects for databases should be disabled.
      Returns:
      whether the mocked instance starts successfully,

    • invokeScheduleTaskWithCustomizedHandling

      public ScheduleTaskInvocationResultDto invokeScheduleTaskWithCustomizedHandling(String scheduleTaskDtos, boolean enabled)
      Description copied from interface: SutHandler

      a method to employ customized mocking for database

      Specified by:
      invokeScheduleTaskWithCustomizedHandling in interface SutHandler
      Parameters:
      scheduleTaskDtos - contains info about schedule tasks should be invoked
      enabled - reflect to enable (set it true) or terminate (set it false) the schedule task
      Returns:
      invocation results of schedule task

    • readFileAsStringFromTestResource

      public final String readFileAsStringFromTestResource(String fileName)
      Parameters:
      fileName - the name of file which exist in the same directory of the class
      Returns:
      content of file with the specified file

    • getExceptionImportanceLevels

      public Map<Class,Integer> getExceptionImportanceLevels()
      Description copied from interface: CustomizationHandler

      specify importance levels for exceptions lower value more important, 0 is the most important exception which needs to be fixed earliest

      Specified by:
      getExceptionImportanceLevels in interface CustomizationHandler
      Returns:
      a map, key is the class of exception, and value is importance level which must not be less than 0