www.espertech.comDocumentation
The Esper compiler and runtime have been developed to address the requirements of applications that analyze and react to events. Some typical examples of applications are:
Business process management and automation (process monitoring, BAM, reporting exceptions)
Finance (algorithmic trading, fraud detection, risk management)
Network and application monitoring (intrusion detection, SLA monitoring)
Sensor network applications (RFID reading, scheduling and control of fabrication lines, air traffic)
What these applications have in common is the requirement to process events (or messages) in real-time or near real-time. This is sometimes referred to as complex event processing (CEP) and event series analysis. Key considerations for these types of applications are throughput, latency and the complexity of the logic required.
High throughput - applications that process large volumes of messages (between 1,000 to 100k messages per second)
Low latency - applications that react in real-time to conditions that occur (from a few milliseconds to a few seconds)
Complex computations - applications that detect patterns among events (event correlation), filter events, aggregate time or length windows of events, join event series, trigger based on absence of events etc.
The EPL compiler and runtime were designed to make it easier to build and extend CEP applications.
More information on CEP can be found at FAQ.
Esper is a language, a language compiler and a runtime environment.
The Esper language is the Event Processing Language (EPL). It is a declarative, data-oriented language for dealing with high frequency time-based event data. EPL is compliant to the SQL-92 standard and extended for analyzing series of events and in respect to time.
The Esper compiler compiles EPL source code into Java Virtual Machine (JVM) bytecode so that the resulting executable code runs on a JVM within the Esper runtime environment.
The Esper runtime runs on top of a JVM. You can run byte code produced by the Esper compiler using the Esper runtime.
The Esper architecture is similar to that of other programming languages that are compiled to JVM bytecode, such as Scala, Clojure and Kotlin for example. Esper EPL however is not an imperative (procedural) programming language.
The Esper language is the Event Processing Language (EPL) designed for Complex Event Processing and Streaming Analytics.
EPL is organized into modules. Modules are compiled into byte code by the compiler. We use the term module for an EPL source code unit.
A module consists of statements. Statements are the declarative code for performing event and time analysis. Most statements are in the form of "select ... from ..."
.
We use the term statement for each unit of declarative code that makes up a module.
Your application receives output from statements via callback or by iterating current results of a statement.
A statement can declare an EPL-object such as listed below:
Event types define stream type information and are added using create schema
or by configuration.
Variables are free-form value holders and are added using create variable
or by configuration.
Named windows are sharable named data windows and are added using create window
.
Tables are sharable organized rows with columns that are simple, aggregation and complex types, and are added using create table
.
Contexts define analysis lifecycle and are added using create context
.
Expressions and Scripts are reusable expressions and are added using create expression
.
Indexes organize named window events and table rows for fast lookup and are added using create index
.
Use access modifiers such as private
, protected
and public
to control access to EPL-objects.
A module can optionally have a module name. The module name has a similar use as the package name or namespace name in a programming language. A module name is used to organize EPL objects and to avoid name conflicts.
When deploying a compiled module the runtime assigns a deployment id to the deployment. The deployment id uniquely identifies a given deployment of a compiled module. A compiled module can be parameterized and deployed multiple times.
A statement always has a statement name. The statement name identifies a statement within a deployed module and is unique within a deployment. The combination of deployment id and statement name uniquely identifies a statement within a runtime.
EPL is type-safe in that EPL does not allow performing an operation on an object that is invalid for that object.
Please add the Esper compiler jar file, the common jar file and the compiler dependencies to the classpath of the program that will be compiling EPL.
The jar files listed here are not required for the runtime except for esper-common-
version.jar
.
Common jar file esper-common-
version.jar
Compiler jar file esper-compiler-
version.jar
ANTLR parser jar file antlr4-runtime-4.7.2.jar
SLF4J logging library slf4j-api-1.7.26.jar
Janino Java compiler janino-3.1.0.jar
and commons-compiler-3.1.0.jar
Optionally, for logging using Log4j, please add slf4j-log4j12-1.7.26.jar
and log4j-1.2.17.jar
to the classpath.
There are no additional jar files required by Esper for using Esper with JSON-formatted event documents.
Optionally, for using Apache Avro, please add esper-common-avro-
version.jar
to the classpath.
Optionally, for using XML XSD schemas for event types, please add esper-common-xmlxsd-
version.jar
to the classpath.
Your application can register an event type to instruct the compiler what the input events look like. When compiling modules the compiler checks the available event type information to determine that the module is valid.
This example assumes that there is a Java class PersonEvent
and each instance of the PersonEvent
class is an event.
It is not necessary to create classes for each event type.
It is not necessary to preconfigure each event type. It is not necessary to set up a Configuration
object.
This step-by-step keeps it simple to get you started.
Our event class for the step-by-step is:
package com.mycompany.myapp; public class PersonEvent { private String name; private int age; public PersonEvent(String name, int age) { this.name = name; this.age = age; } public String getName() { return name; } public int getAge() { return age; } }
The different event representations are discussed at Section 3.5, “Comparing Event Representations”.
For declaring event types using EPL with create schema
, see Section 5.15, “Declaring an Event Type: Create Schema”.
Your application can obtain a compiler calling the getCompiler
static method of the EPCompilerProvider
class:
EPCompiler compiler = EPCompilerProvider.getCompiler();
The step-by-step provides a Configuration
object to the compiler that adds the predefined person event:
Configuration configuration = new Configuration(); configuration.getCommon().addEventType(PersonEvent.class);
The sample module for this getting-started section simply has one statement that selects the name and the age of each arriving person event.
It specifies a statement name using the @name
annotation and assigns a name my-statement
to the statement.
@name('my-statement') select name, age from PersonEvent
Compile a module by using the compile
method passing the configuration as part of the compiler arguments:
CompilerArguments args = new CompilerArguments(configuration); EPCompiled epCompiled; try { epCompiled = compiler.compile("@name('my-statement') select name, age from PersonEvent", args); } catch (EPCompileException ex) { // handle exception here throw new RuntimeException(ex); }
Upon compiling this module, the compiler verifies that PersonEvent
exists since it is listed in the from
-clause.
The compiler also verifies that the name
and age
properties are available for the PersonEvent
since they are listed in the select
-clause.
The compiler generates byte code for extracting property values and producing output events.
The compiler builds internal data structures for later use by filter indexes to ensure that when a PersonEvent
comes in it will be processed fast.
More information on the compile API can be found at Chapter 15, Compiler Reference and the JavaDoc.
Please add the Esper common jar file, the runtime jar file and the runtime dependencies to the classpath of the program that will be executing compiled modules. The runtime jar file is not required for the compiler.
Common jar file esper-common-
version.jar
Runtime jar file esper-runtime-
version.jar
SLF4J logging library slf4j-api-1.7.25.jar
Optionally, for logging using Log4j, please add slf4j-log4j12-1.7.25.jar
and log4j-1.2.17.jar
to the classpath.
There are no additional jar files required by Esper for using Esper with JSON-formatted event documents.
Optionally, for using Apache Avro, please add esper-common-avro-
version.jar
to the classpath.
Optionally, for using XML XSD schemas for event types, please add esper-common-xmlxsd-
version.jar
to the classpath.
The step-by-step provides a Configuration
object to the runtime that adds the predefined person event:
Configuration configuration = new Configuration(); configuration.getCommon().addEventType(PersonEvent.class);
It is not necessary to preconfigure each event type. It is not necessary to set up a Configuration
object.
For this example however since the compiler knows PersonEvent
as a predefined type and it must this be preconfigured for the runtime as well.
Your application can obtain a runtime by calling the getDefaultRuntime
static method of the EPRuntimeProvider
class and passing the configuration:
EPRuntime runtime = EPRuntimeProvider.getDefaultRuntime(configuration);
More information about the runtime can be found at Chapter 16, Runtime Reference and the JavaDoc.
More information about configuration can be found at Chapter 17, Configuration and the JavaDoc.
Your application can deploy a compiled module using the deploy
method of the administrative interface.
The API calls are:
EPDeployment deployment; try { deployment = runtime.getDeploymentService().deploy(epCompiled); } catch (EPDeployException ex) { // handle exception here throw new RuntimeException(ex); }
As part of deployment, the runtime verifies that all module dependencies, such as event types, do indeed exist.
During deployment the runtime adds entries to filter indexes to ensure that when a PersonEvent
comes in it will be processed fast.
Your application can attach a callback to the EPStatement
to receive statement results. The following sample callback simply prints name and age:
EPStatement statement = runtime.getDeploymentService().getStatement(deployment.getDeploymentId(), "my-statement"); statement.addListener( (newData, oldData, statement, runtime) -> { String name = (String) newData[0].get("name"); int age = (int) newData[0].get("age"); System.out.println(String.format("Name: %s, Age: %d", name, age)); });
Your application can provide different kinds of callbacks, see Table 16.2, “Choices For Receiving Statement Results”.
Your application can send events into the runtime using the sendEventBean
method (or other sendEvent
method matching your choice of event) that is part of the runtime interface:
runtime.getEventService().sendEventBean(new PersonEvent("Peter", 10), "PersonEvent");
The output you should see is:
Name: Peter, Age: 10
Upon sending the PersonEvent
event object to the runtime, the runtime consults the internally-maintained shared filter index tree structure to determine if any statement is interested in PersonEvent
events.
The statement that was deployed as part of this example has PersonEvent
in the from
-clause, thus the runtime delegates processing of such events to the statement.
The compiled bytecode obtains the name and age properties by calling the getName
and getAge
methods.
The compiler and runtime both require the following 3rd-party libraries:
SLF4J is a logging API that can work together with LOG4J and other logging APIs. While SLF4J is required, the LOG4J log component is not required and can be replaced with other loggers. SLF4J is licensed under Apache 2.0 license as provided in lib/esper_3rdparties.license
.
The compiler requires the following 3rd-party libraries for compiling only (and not at runtime):
ANTLR is the parser generator used for parsing and parse tree walking of the pattern and EPL syntax. Credit goes to Terence Parr at http://www.antlr.org. The ANTLR license is a BSD license and is provided in lib/esper_3rdparties.license
. The antlr-runtime
runtime library is required for runtime.
Janino is a small and fast Java compiler. The compiler generates code and compiles generated code using Janino. Janino is licensed under 3-clause New BSD License as provided in lib/esper_3rdparties.license
.