www.espertech.comDocumentation
This chapter presents information related to the development lifecycle for developing an event processing application with EPL. It includes information on authoring, testing, debugging, packaging and deploying.
Enterprise Edition includes authoring tools for EPL statements and modules by providing form-based dialogs, templates, an expression builder, simulation tool and other tools. Enterprise Edition also supports hot deployment and packaging options for EPL and related code.
EPL statements can be organized into modules as described below. Any text editor can edit EPL statement and module text. A text editor or IDE that highlights SQL syntax or keywords works.
For authoring configuration files please consult the XSD schema files as provided with the distribution.
For information on authoring event classes or event definitions in general please see Chapter 3, Event Representations or Section 5.15, “Declaring an Event Type: Create Schema”.
We recommend testing EPL statements using a test framework such as JUnit or TestNG. Please consult the Esper test suite for extensive examples, which can be downloaded from the distribution site.
Esper's API provides test framework classes to simplify automated testing of EPL statements. Please see Section 14.20, “Test and Assertion Support” for more information.
We recommend performing latency and throughput tests early in the development lifecycle. Please consider the performance tips in Chapter 23, Performance for optimal performance.
Consider engine and statement metrics reporting for identifying slow-performing statements, for example. See Section 14.14, “Engine and Statement Metrics Reporting”.
Enterprise Edition includes a debugger for EPL statement execution.
One important tool for debugging without Enterprise Edition is the parameterized @Audit
annotation. This annotation allows to output, on statement-level, detailed information about many aspects of statement processing.
Another tool for logging engine-level detail is Section 15.4.16.1, “Execution Path Debug Logging”.
Please see Section 15.7, “Logging Configuration” for information on configuring logging in general.
Use the @Audit
annotation to have the engine output detailed information about statement processing. The engine reports, at INFO level, the information under log name com.espertech.esper.audit
. You may define an output format for audit information via configuration.
You may provide a comma-separated list of category names to @Audit
to output information related to specific categories only. The table below lists all available categories. If no parameter is provided, the engine outputs information for all categories. Category names are not case-sensitive.
For the next statement the engine produces detailed processing information (all categories) for the statement:
@Name('All Order Events') @Audit select * from OrderEvent
For the next statement the engine provides information about new events and also about event property values (2 categories are listed):
@Name('All Order Events') @Audit('stream,property') select price from OrderEvent
Here is a more complete example that uses the API to create the schema, create above statement and send an event:
epService.getEPAdministrator().createEPL("create schema OrderEvent(price double)"); String epl = "@Name('All-Order-Events') @Audit('stream,property') select price from OrderEvent"; epService.getEPAdministrator().createEPL(epl).addListener(listener); epService.getEPRuntime().sendEvent(Collections.singletonMap("price", 100d), "OrderEvent");
The output is similar to the following:
INFO [audit] Statement All-Order-Events stream OrderEvent inserted {price=100.0} INFO [audit] Statement All-Order-Events property price value 100.0
Table 16.1. @Audit Categories
Category | Description |
---|---|
ContextPartition | Each context partition allocation and de-allocation (only for statements that declare a context). |
Dataflow-Source | Each data flow source operator providing an event. |
Dataflow-Op | Each data flow operator processing an event. |
Dataflow-Transition | Each data flow instance state transition. |
Exprdef | Each expression declaration name and return value. |
Expression | Each top-level expression and its return value. |
Expression-nested | Each expression including child or nested expressions and their return value. |
Insert | Each event inserted via insert-into. |
Pattern | Each pattern sub-expression and its change in truth-value. |
Pattern-instances | Each pattern sub-expression and its count of active instances. |
Property | Each property name and the event's property value. |
Schedule | Each schedule modification and trigger received by a statement. |
Stream | Each new event received by a statement. |
View | Each data window name and its insert and remove stream. |
Note that the engine only evaluates select-clause expressions if either a listener or subscriber is attached to the statement or if used with insert-into.
Please consider Esper Enterprise Edition as a target deployment platform. Esper alone does not ship with a server as it is designed as a core CEP engine.
For un-deploying, deploying or re-deploying single or multiple statements or modules as an atomic management unit please see Section 14.3.7, “Atomic Statement Management”.
To support packaging and deploying event-driven applications, Esper offers infrastructure as outlined herein:
EPL modules to build a cohesive, easily-externalizable deployment unit out of related statements as described in Section 16.5, “EPL Modules”.
The deployment administrative interface is described in Section 16.6, “The Deployment Administrative Interface”.
Instructions and code for use when the deployment target is a J2EE web application server or servlet runtime, please see Section 16.7, “J2EE Packaging and Deployment”.
An EPL module file is a plain text file in which EPL statements appear separated by the semicolon (;) character. It bundles EPL statements with optional deployment instructions. A service provider instance keeps track of the known and/or deployed EPL modules and makes it easy to add, remove, deploy and un-deploy EPL modules.
The synopsis of an EPL module file is:
[module module_name;] [uses module_name; | import import_name;] [uses module_name; | import import_name;] [...] [epl_statement;] [epl_statement;] [...]
Use the module
keyword followed a module_name identifier or a package (identifiers separated by dots) to declare the name of the module. The module name declaration must be at the beginning of the file, comments and whitespace excluded. The module name
serves to check uses-dependences of other modules.
If a module file requires certain constructs that may be shared by other module files, such as named windows, tables, variables, event types, variant streams or inserted-into streams required by statements,
a module file may specify zero to many dependent modules with the uses
keyword. At deployment time the engine checks the uses-dependencies
and ensures that a module of that name is already deployed or will be deployed as part of the deployments. The deployment API supports ordering modules according to their uses-relationship.
If the EPL statements in the module require Java classes such as for underlying events or user-defined functions, use the import
keyword followed by the fully-qualified class name or package name in the format package.*
.
The uses
and import
keywords are optional and must occur after the module
declaration.
Following the optional deployment instructions are any number of epl_statement EPL statements that are separated by semicolon (;
).
The following is a sample EPL module file explained in detail thereafter:
// Declare the name for the module module org.myorganization.switchmonitor; // Declare other module(s) that this module depends on uses org.myorganization.common; // Import any Java/.NET classes in an application package import org.myorganization.events.*; // Declare an event type based on a Java class in the package that was imported as above create schema MySwitchEvent as MySwitchEventPOJO; // Sample statement @Name('Off-On-Detector') insert into MyOffOnStream select * from pattern[every-distinct(id) a=MySwitchEvent(status='off') -> b=MySwitchEvent(id=a.id, status='on')]; // Sample statement @Name('Count-Switched-On') @Description('Count per switch id of the number of Off-to-On switches in the last 1 hour') select id, count(*) from MyOffOnStream#time(1 hour) group by id;
The example above declares a module name of org.myorganization.switchmonitor
. As defined by the uses
keyword, it ensures that the org.myorganization.common
module is already deployed.
The example demonstrates the import
keyword to make a package name known to the engine for resolving POJO class names, as the example assumes that MySwitchEventPOJO
is a POJO event class.
In addition the example module contains two statements separated by semicolon characters.
Your application code may, after deployment, look up a statement and attach listeners as shown here:
epService.getEPAdministrator().getStatement("Count-Switched-On").addListener(...);
The com.espertech.esper.client.deploy.EPDeploymentAdmin
service available from the EPAdministrator
interface by method getDeploymentAdmin
provides the functionality available to manage packaging and deployment.
Please consult the JavaDoc documentation for more information.
The deployment API allows to read resources and parse text strings to obtain an object representation of the EPL module, the Module
. A Module
object can also be simply constructed.
After your application obtains a Module
instance it may either use deploy
to deploy the module directly, starting all statements of the module. Alternatively your application
may add a module, making it known without starting statements for later deployment. In each case the module is assigned a deployment id, which acts as a unique primary key for all known modules. Your application may assign its own deployment id
or may have the engine generate a deployment id (two footprints for add
and deploy
methods).
A module may be in two states: un-deployed or deployed. When calling add
to add a module, it starts life in the un-deployed state. When calling deploy
to deploy a module, it starts life in the deployed state.
A module may be transitioned by providing the deployment id and by calling the deploy
or undeploy
methods.
Your code can remove a module in un-deployed state using the remove
method or the undeployRemove
method. If the module is in deployed state, use undeployRemove
to un-deploy and remove the module.
The DeploymentOptions
instance that can be passed to the deploy
method when validating or deploying modules controls validation, fail-fast, rollback and the isolated service provider, if any, for the deployment.
Also use DeploymentOptions
to set a user object per statement and to set a statement name per statement.
Deployment and un-deployment operations are, by default, atomic operations: Events that come in during deployment or un-deployment are processed after deployment or un-deployment completed.
Using DeploymentOptions
you may provide a strategy for obtaining and releasing the engine-wide lock.
Please also see Section 14.3.7, “Atomic Statement Management” for more information.
We provide additional sample code to read and deploy modules as part of the J2EE considerations below.
Read and parse module files via the EPDeploymentAdmin
interface read
and parse
methods, which returns a Module
instance to represent the module information.
This code snippet demonstrates reading and parsing a module given a file name:
EPServiceProvider epService = EPServiceProviderManager.getDefaultProvider(); EPDeploymentAdmin deployAdmin = epService.getEPAdministrator().getDeploymentAdmin(); Module module = deployAdmin.read(new File("switchmonitor.epl"));
The service provides additional read and parse methods to read from a URL, classpath, input stream or string.
Since modules may have inter-dependencies as discussed under the uses
declaration, the deployment interface provides the getDeploymentOrder
method to order a collection of modules before deployment.
Assuming your application reads multiple modules into a mymodules
module list, this code snippet orders the modules for deployment and validates dependency declarations for each module:
List<Module> mymodules = ... read modules...; DeploymentOrder order = deployAdmin.getDeploymentOrder(mymodules, new DeploymentOrderOptions());
The deployment interface returns a deployment id for each module made known by adding a module or by deploying a module. To un-deploy the module your application must provide the deployment id. Your application can assign its own deployment id or obtain the module name
from the Module
and use that as the deployment id.
The undeploy
operation removes all named windows, tables, variables, event types or any other information associated to the statements within the module to be un-deployed.
The next code snippet deploys modules, starting each modulle's EPL statements:
for (Module mymodule : order.getOrdered()) { DeploymentResult deployResult = deployAdmin.deploy(mymodule, new DeploymentOptions()); }
Un-deploying a module destroys all started statements associated to the module.
To un-deploy and at the same time remove the module from the list of known modules use the undeployRemove
method and pass the deployment id:
deployAdmin.undeployRemove(deployResult.getDeploymentId());
The deployment interface returns all module information that allows your application to determine which modules are known and their current state.
To obtain a list of all known modules or information for a specific module, the calls are:
DeploymentInformation[] info = deployAdmin.getDeploymentInformation(); // Given a deployment id, return the deployment information DeploymentInformation infoModule = deployAdmin.getDeploymen(deploymentId);
The following sample code adds a module, transitions the module to deployed, then un-deploys and removes the module entirely;
// This sample uses the parse method to obtain a module Module module = deployAdmin.parse("create schema MySchema (col1 int)"; // Make the module know; It now shows up in un-deployed state String moduleDeploymentId = deployAdmin.add(module); // Start all statements, passing a null options object for default options deployAdmin.deploy(moduleDeploymentId, null); // Un-deploy module, destroying all statements deployAdmin.undeploy(moduleDeploymentId); // Remove module; It will no longer be known deployAdmin.remove(moduleDeploymentId);
Use the @Name
annotation to assign a name to each statement that your application would like to attach a listener or subscriber, or look up the statement for iteration or management by the administrative API.
Use the create schema
syntax and the import
keyword to define event types. When sharing event types, named windows or variables between modules use the uses
keyword to
declare a separate module that holds the shared definitions.
To validate whether a set of statements is complete and can start without issues, set the following flags on a DeploymentOptions
instance passed to the deploy
method as the code snippet below shows:
DeploymentOptions options = new DeploymentOptions(); options.setIsolatedServiceProvider("validation"); // isolate any statements options.setValidateOnly(true); // validate leaving no started statements options.setFailFast(false); // do not fail on first error epService.getEPAdministrator().getDeploymentAdmin() .deploy(module, options);
Esper can well be deployed as part of a J2EE web or enterprise application archive to a web application server. When designing for deployment into a J2EE web application server, please consider the items discussed here.
We provide a sample servlet context listener in this section that uses the deployment API to deploy and un-deploy modules as part of the servlet lifecycle.
The distribution provides a message-driven bean (MDB) example that you may find useful.
Esper does not have a dependency on any J2EE or Servlet APIs to allow the engine to run in any environment or container.
As multiple web applications deployed to a J2EE web application server typically have a separate classloader per application, you should consider whether engine instances need to be shared between applications or can remain separate engine instances. Consider the EPServiceProviderManager
a Singleton.
When deploying multiple web applications, your J2EE container classloader may provide a separate instance of the Singleton EPServiceProviderManager
to each web application resulting in multiple independent engine instances.
To share EPServiceProvider
instances between web applications, one approach is to add the Esper jar files to the system classpath. A second approach can be to have multiple web applications share the same servet context and have your application place the EPServiceProvider
instance into a servlet context attribute for sharing. Architecturally you may also consider a single archived application (such as an message-driven bean) that all your web applications communicate to via the JMS broker provided by your application server or an external JMS broker.
As per J2EE standards there are restrictions in regards to starting new threads in J2EE application code. Esper adheres to these restrictions: It allows to be driven entirely by external events. To remove all Esper threads, set the internal timer off and leave the advanced threading options turned off. To provide timer events when the internal timer is turned off, you should check with your J2EE application container for support of the Java system timer or for support of batch or work loading to send timer events to an engine instance.
As per J2EE standards there are restrictions in regards to input and output by J2EE application code. Esper adheres to these restrictions: By itself it does not start socket listeners or performs any file IO.
When deploying a J2EE archive that contains EPL modules files below is sample code to read and deploy EPL modules files packaged with the enterprise or web application archive when the servlet initializes. The sample un-deploys EPL modules when the servlet context gets destroyed.
A sample web.xml
configuration extract is:
<?xml version="1.0" encoding="UTF-8"?> <web-app> <listener> <listener-class>SampleServletListener</listener-class> </listener> <context-param> <param-name>eplmodules</param-name> <param-value>switchmonitor.epl</param-value> </context-param> </web-app>
A servet listener that deploys EPL module files packaged into the archive on context initialization and that un-deploys when the application server destroys the context is shown here:
public class SampleServletListener implements ServletContextListener { private List<String> deploymentIds = new ArrayList<String>(); public void contextInitialized(ServletContextEvent servletContextEvent) { try { EPServiceProvider epServiceProvider = EPServiceProviderManager.getDefaultProvider(); String modulesList = servletContextEvent.getServletContext().getInitParameter("eplmodules"); List<Module> modules = new ArrayList<Module>(); if (modulesList != null) { String[] split = modulesList.split(","); for (int i = 0; i < split.length; i++) { String resourceName = split[i].trim(); if (resourceName.length() == 0) { continue; } String realPath = servletContextEvent.getServletContext().getRealPath(resourceName); Module module = epServiceProvider.getEPAdministrator() .getDeploymentAdmin().read(new File(realPath)); modules.add(module); } } // Determine deployment order DeploymentOrder order = epServiceProvider.getEPAdministrator() .getDeploymentAdmin().getDeploymentOrder(modules, null); // Deploy for (Module module : order.getOrdered()) { DeploymentResult result = epServiceProvider.getEPAdministrator() .getDeploymentAdmin().deploy(module, new DeploymentOptions()); deploymentIds.add(result.getDeploymentId()); } } catch (Exception ex) { ex.printStackTrace(); } } public void contextDestroyed(ServletContextEvent servletContextEvent) { EPServiceProvider epServiceProvider = EPServiceProviderManager.getDefaultProvider(); for (String deploymentId : deploymentIds) { epServiceProvider.getEPAdministrator().getDeploymentAdmin().undeployRemove(deploymentId); } } }
The engine can report key processing metrics through the JMX platform mbean server by setting a single configuration flag described in Section 15.4.23, “Engine Settings Related to JMX Metrics”.
Engine and statement-level metrics reporting is described in Section 14.14, “Engine and Statement Metrics Reporting”.
The easiest way to see thread contentions is by using VisualVM when Esper is under load and looking at the Threads tab. In the worst case you will see a lot of red color in VisualVM. The red line in VisualVM shows the threads that are either in a monitor region or waiting in an entry set for the monitor. The monitor is the mechanism that Java uses to support synchronization. When an EPL statement is stateful, Esper manages the state using a monitor (lock) per context partition.
A JVM profiler can be handy to see how much CPU is spent in Esper by the sendEvent
method.
The jconsole
can provide information on the JVM heap. If memory gets tights the performance can drop significantly.
Enterprise Edition has a library for measuring and reporting memory use for Esper.