Decision Engine configuration requirements

Definitions

In this document, we separate the Decision Engine (DE) initialization into two phases:

• Configuration: The reading and processing of user-specified parameters that influence the run-time behavior of the program, and their subsequent preparation for use during the construction phase.
• Construction: The formation of valid in-memory objects based on parameters provided from the configuration stage.

The configuration system refers to the collection of utilities used for the configuration stage, as defined above. Important components of the configuration system include the:

1. Configuration language
2. In-memory representation of the parameters and their corresponding values
3. Interface for the framework and module code that must access the configuration parameters
4. Infrastructure responsible for recording and persisting configuration history
5. Infrastructure or procedures used to validate user-supplied configuration artifacts
6. Configuration documentation facility to aid operators and developers in understanding how to successfully configure various components of the decision-engine system or cycle
7. Tools available for inspecting and debugging configurations

In what follows, to reconfigure a DE component means to apply a configuration change to an object that is already active in either the DE application itself or a decision channel.

Current configuration system

The configuration of the DE system can be broken into three categories: the DE application itself, a DE channel, and a module used in the channel. See below for the supported configurations (heuristically) of each category. At the moment, in order to change the configuration of a DE channel (and its modules), either the DE application must be restarted, or the channels must be reloaded by means of a signal handler, registered in the DE constructor.

DE application

decision_engine: {
  server_address: (address_string, port_number) # for internet protocol
  logger: {
    log_file: file_name_str
    max_file_size: num_in_bytes # default is 200 MB
    max_backup_count: num # default is 6
  }
  dataspace: {
    datasource: {
      name: ""
      module: ""
      config: ""
    }
  }
}

DE channel

channel: {
  sources: {}
  publishers: {}
  logicengines: {}
  transforms: {}
  
  # Optional parameters
  task_manager: { 
    data_TO: timeout_in_seconds # default is 300 (5 min.)
  }
}

DE module

module_name: {
  name: ""
  module: ""
  schedule: num # default is 300 (5 min.)
  parameters: {
    ...
  }
}

Configuration system requirements

Basic attributes:

1. Each configurable DE component (e.g. framework, user module, monitoring plugin) shall be configurable through a human-readable, text file.
2. A configuration parameter shall consist of a value bound to a name.
3. The supported value types shall be:
   1. atomic: values with no underlying structure (e.g character strings, numbers, etc.), or
   2. composite: values that can be represented by either a list of other values, or a collection of configuration parameters.
4. It shall be possible to specify a name within a named, enclosing scope; the named, enclosing scope is thus a parameter with a composite value.
5. A parameter named in one scope shall be distinct from a parameter named in a different scope, even if the parameter name is the same.

To avoid duplicate configuration specifications:

6. Within a configuration file, it shall be possible to substitute the value of a previously named parameter within that same file.
7. Within a configuration file, it shall be possible to refer to or reuse values from other configuration files (hereafter referenced configuration files).
8. A lookup policy shall be specified to locate referenced configuration files.

On default values:

9. The configuration system shall support default values that can be overriden by an explicit parameter assignment.
10. It shall be possible to easily ascertain what the default configuration values are.

System-wide:

11. The configuration system shall make a fully processed configuration available on demand for parameter inspection.
12. The configuration system shall have the ability to provide diagnostic information as to the source of the final parameter values of a fully processed configuration.
13. A fully processed configuration shall not be modified/modifiable by the DE framework or by any DE components (such as modules, monitoring infrastructure, etc.).
14. It must be clear to an operator which configuration changes require a restart of the DE application.

• e.g. Upon reconfiguring a DE workflow component that requires a restart of the DE application, a confirm request shall be asked of the operator.

15. Any changes to a decision-channel configuration shall not require a restart of the DE application.
16. It shall be possible to exclude resource types without changing the code of the DE framework, module, or any other DE component.
17. To the extent possible, reconfiguring a DE component shall not influence any logically separate, yet active DE components.
18. To the extent possible, reconfiguring a DE component shall not require a restart of the DE application, with the exception of reconfiguring the application itself.
19. Enough configuration history shall be archived to replay a decision cycle.

Note that the configuration system meeting the requirements above is not required to support any configuration artifacts that pre-date it.

Configuration system preferences

The following items, while desirable, are not strictly necessary in realizing the configuration system design:

1. A fully processed configuration for the DE application or a DE channel should be representable as a valid configuration file.
2. The configuration language should not be sensitive to whitespace differences (e.g. Python is a poor match).
3. For backwards compatibility reasons, it is preferable if the Python configuration object passed to a DE component is similar in interface to a Python dictionary.
4. The configuration should be versioned, and that version should be recorded in the data block.
5. The system should be able to print out what configuration parameters a given module supports.
6. The system should be able to print out a valid configuration snippet based on the parameters supported by a given module.
7. Various tools should be provided:
   1. A validation mechanism to ensure that a configuration file is valid before it is scheduled for use.
   2. Utilities to determine how configuration files relate to one another.
   3. A view into the actual configuration data without having to understand the representation used in recording or persisting the configuration.
8. The syntax of the configuration language should either be supported or be easily supportable by most text editors.

Related issues

Various issues were raised related to the use of the configuration system:

1. A consistent naming convention should be chosen or recommended for any DE-supported configuration (“decisionengine” vs. “decision_engine” vs. “decision-engine”, etc.).
2. Any transform code with parameters that may need to be tweaked should expose those parameters in the corresponding configuration file.
3. Only a minimum set of parameters should be required to start a DE application.
4. The logging and monitoring subsystems must be configurable, as well as module-specific behavior (e.g. the number of retries allowed).

Related to usability:

5. It would be desirable to have an editor that provides a user-friendly interface for editing configuration files. The editor could provide a graphical interface for adjusting parameters, with actual parameter assignments happening under-the-covers.

Related to the logic engine:

6. Although configurable, the types of expressions allowed for logic-engine facts and rules are not part of the general configuration system.