Package com.yahoo.searchdefinition.processing
Classes in this package (processors) implements some search
definition features by reducing them to simpler features.
The processors are run after parsing of the search definition,
before creating the derived model.
For simplicity, features should always be implemented here
rather than in the derived model if possible.
New processors must be added to the list in Processing.
-
Interface Summary Interface Description Processing.ProcessorFactory -
Class Summary Class Description AddAttributeTransformToSummaryOfImportedFields Adds the attribute summary transform (SummaryTransform.ATTRIBUTE
to allSummaryField
having an imported field as source.AddExtraFieldsToDocument This processor creates aSDDocumentType
for eachSchema
object which holds all the data that search associates with a document described in a search definition file.AdjustPositionSummaryFields AttributeProperties Checks that attribute properties only are set for attributes that have data (are created by an indexing statement).AttributesImplicitWord Fields that derive to attribute(s) and no indices should use the WORD indexing form, in a feeble attempt to match the most peoples expectations as closely as possible.Bolding Checks that bolding or dynamic summary is turned on only for text fields.BoolAttributeValidator Validates attribute fields using bool type, ensuring the collection type is supported.BuiltInFieldSets Adds field sets for 1) fields defined inside document type 2) fields inside search but outside documentCreatePositionZCurve Adds a "fieldName_zcurve" long attribute and "fieldName.distance" and "FieldName.position" summary fields to all position type fields.DictionaryProcessor Propagates dictionary settings from field level to attribute level.DisallowComplexMapAndWsetKeyTypes Non-primitive key types for map and weighted set forbidden (though OK in document model)DiversitySettingsValidator ExactMatch The implementation of exact matchingFastAccessValidator Validates the use of the fast-access property.FieldSetSettings Computes the right "index commands" for each fieldset in a search definition.FilterFieldNames Takes the fields and indexes that are of type rank filter, and stores those names on all rank profilesImplicitSummaries Makes implicitly defined summaries into explicit summariesImplicitSummaryFields This processor adds all implicit summary fields to all registered document summaries.ImportedFieldsResolver Iterates all imported fields from schema parsing and validates and resolves them into concrete fields from referenced document types.IndexFieldNames Because of the way the parser works (allowing any token as identifier), it is not practical to limit the syntax of field names there, do it here.IndexingInputs This processor modifies all indexing scripts so that they input the value of the owning field by default.IndexingOutputs This processor modifies all indexing scripts so that they output to the owning field by default.IndexingValidation IndexingValues IntegerIndex2Attribute Replaces the 'index' statement of all numerical fields to 'attribute' because we no longer support numerical indexes.LiteralBoost Expresses literal boosts in terms of extra indices with rank boost.MakeAliases Takes the aliases set on field by parser and sets them on correct Index or AttributeMakeDefaultSummaryTheSuperSet All summary fields which are not attributes must currently be present in the default summary class, since the default summary class also defines the docsum.dat format.MatchConsistency Warn on inconsistent match settings for any indexMatchedElementsOnlyResolver Iterates all summary fields with 'matched-elements-only' and adjusts transform (if all struct-fields are attributes) and validates that the field type is supported.MatchPhaseSettingsValidator Validates the match phase settings for all registered rank profiles.MatchPhaseSettingsValidator.AttributeValidator MultifieldIndexHarmonizer Ensures that there are no conflicting types or field settings in multifield indices, either by changing settings or by splitting conflicting fields in multiple ones with different settings.MutableAttributes NGramMatch The implementation of "gram" matching - splitting the incoming text and the queries into n-grams for matching.OnnxModelConfigGenerator Processes ONNX ranking features of the form: onnx("files/model.onnx", "path/to/output:1") And generates an "onnx-model" configuration as if it was defined in the schema: onnx-model files_model_onnx { file: "files/model.onnx" } Inputs and outputs are resolved in OnnxModelTypeResolver, which must be processed after this.OnnxModelTypeResolver Processes every "onnx-model" element in the schema.OptimizeIlscript Run ExpressionOptimizer on all scripts, to get rid of expressions that have no effect.PagedAttributeValidator Validates the 'paged' attribute setting and throws if specified on unsupported types.PredicateProcessor Validates the predicate fields.Processing Executor of processors.Processor Abstract superclass of all search definition processors.RankingExpressionTypeResolver Resolves and assigns types to all functions in a ranking expression, and validates the types of all ranking expressions under a search instance: Some operators constrain the types of inputs, and first-and second-phase expressions must return scalar values.ReferenceFieldsProcessor Class that processes reference fields and removes attribute aspect of such fields from summary.ReservedDocumentNames ReservedFunctionNames Issues a warning if some function has a reserved name.SearchMustHaveDocument A search must have a document definition of the same name inside of it, otherwise crashes may occur as late as during feedingSetRankTypeEmptyOnFilters All rank: filter fields should have rank type empty.SortingSettings Validate conflicting settings for sortingStringSettingsOnNonStringFields SummaryConsistency Ensure that summary field transforms for fields having the same name are consistent across summary classesSummaryDiskAccessValidator Emits a warning for summaries which accesses disk.SummaryDynamicStructsArrays Fail if: An SD field explicitly says summary:dynamic , but the field is wset, array or struct.SummaryFieldsMustHaveValidSource Verifies that the source fields actually refers to a valid field.SummaryNamesFieldCollisions Verifies that equally named summary fields in different summary classes don't use different fields for source.TagType The implementation of the tag datatypeTensorFieldProcessor Class that processes and validates tensor fields.TextMatch TypedTransformProvider UriHack UrlFieldValidator ValidateFieldTypes This Processor makes sure all fields with the same name have the sameDataType
.ValidateFieldTypesDocumentsOnly ValidateFieldWithIndexSettingsCreatesIndex Check that fields with index settings actually creates an index or attributeValidateStructTypeInheritance WordMatch The implementation of word matching - with word matching the field is assumed to contain a single "word" - some contiguous sequence of word and number characters - but without changing the data at the indexing side (as with text matching) to enforce this.