Esper Reference

The Esper compiler and runtime have been developed to address the requirements of applications that analyze and react to events. Some typical examples of applications are:

What these applications have in common is the requirement to process events (or messages) in real-time or near real-time. This is sometimes referred to as complex event processing (CEP) and event series analysis. Key considerations for these types of applications are throughput, latency and the complexity of the logic required.

The EPL compiler and runtime were designed to make it easier to build and extend CEP applications.

More information on CEP can be found at FAQ.

Esper is a language, a language compiler and a runtime environment.

The Esper language is the Event Processing Language (EPL). It is a declarative, data-oriented language for dealing with high frequency time-based event data. EPL is compliant to the SQL-92 standard and extended for analyzing series of events and in respect to time.

The Esper compiler compiles EPL source code into Java Virtual Machine (JVM) bytecode so that the resulting executable code runs on a JVM within the Esper runtime environment.

The Esper runtime runs on top of a JVM. You can run byte code produced by the Esper compiler using the Esper runtime.

The Esper architecture is similar to that of other programming languages that are compiled to JVM bytecode, such as Scala, Clojure and Kotlin for example. Esper EPL however is not an imperative (procedural) programming language.

The Esper language is the Event Processing Language (EPL) designed for Complex Event Processing and Streaming Analytics.

EPL is organized into modules. Modules are compiled into byte code by the compiler. We use the term module for an EPL source code unit.

A module consists of statements. Statements are the declarative code for performing event and time analysis. Most statements are in the form of "select ... from ...". We use the term statement for each unit of declarative code that makes up a module.

Your application receives output from statements via callback or by iterating current results of a statement.

A statement can declare an EPL-object such as listed below:

Use access modifiers such as private, protected and public to control access to EPL-objects.

A module can optionally have a module name. The module name has a similar use as the package name or namespace name in a programming language. A module name is used to organize EPL objects and to avoid name conflicts.

When deploying a compiled module the runtime assigns a deployment id to the deployment. The deployment id uniquely identifies a given deployment of a compiled module. A compiled module can be parameterized and deployed multiple times.

A statement always has a statement name. The statement name identifies a statement within a deployed module and is unique within a deployment. The combination of deployment id and statement name uniquely identifies a statement within a runtime.

EPL is type-safe in that EPL does not allow performing an operation on an object that is invalid for that object.

For NEsper .NET also see Section J.11, “.NET Basic Concepts”.

Statements are continuous queries that analyze events and time and that detect situations.

You interact with Esper by compiling and deploying modules that contain statements, by sending events and advancing time and by receiving output by means of callbacks or by polling for current results.

Figure 2.1. Runtime is a Statement Container

Statements can be partitioned. A partitioned statement can have multiple partitions. For example, there could be partition for each room in a building. For a building with 10 rooms you could have one statement that has 10 partitions. Please refer to Chapter 4, Context and Context Partitions.

A statement that is not partitioned implicitly has one partition. Upon deploying the un-partitioned statement the runtime allocates the single partition. Upon undeploying the un-partitioned statement the runtime destroys the partition.

A partition (or context partition) is where the runtime keeps the state. In the picture above there are three un-partitioned statement and one partitioned statement that has three partitions.

The next sections discuss various easily-understood statements. The sections illustrate how statements behave, the information that the runtime passes to callbacks (the output) and what information the runtime remembers for statements (the state, all state lives in a partition). The sample statements assume an event type by name Withdrawal that has account and amount properties.

This statement selects all Withdrawal events.

select * from Withdrawal

Upon a new Withdrawal event arriving, the runtime passes the arriving event, unchanged and the same object reference, to callbacks. After that the runtime effectively forgets the current event.

The diagram below shows a series of Withdrawal events (1 to 6) arriving over time. In the picture the W_n stands for a specific Withdrawal event arriving. The number in parenthesis is the withdrawal amount.

Figure 2.2. EPL Select Example Input and Output Flow

An aggregation function is a function that groups multiple events together to form a single value. Please find more information at Section 10.2, “Aggregation Functions”.

This statement selects a count and a total amount of all Withdrawal events.

select count(*), sum(amount) from Withdrawal

Upon a new Withdrawal event arriving, the runtime increments the count and adds the amount to a running total. It passes the new count and total to callbacks. After that the runtime effectively forgets the current event and does not remember any events at all, but does remember the current count and total.

Figure 2.3. EPL Select Example Input and Output Flow

Place filter expressions in parenthesis after the event type name. For further information see Section 5.4.1, “Filter-Based Event Streams”.

This statement selects Withdrawal events that have an amount of 200 or higher:

select * from Withdrawal(amount >= 200)

Upon a new Withdrawal event with an amount of 200 or higher arriving, the runtime passes the arriving event to callbacks.

Figure 2.4. EPL Filter Example Input and Output Flow

This statement selects the count and the total amount for Withdrawal events that have an amount of 200 or higher:

select count(*), sum(amount) from Withdrawal(amount >= 200)

Upon a new Withdrawal event with an amount of 200 or higher arriving, the runtime increments the count and adds the amount to the running total. The runtime passes the count and total to callbacks.

Figure 2.5. EPL Filter and Aggregation Example Input and Output Flow

A data window, or window for short, retains events for the purpose of aggregation, join, match-recognize patterns, subqueries, iterating via API and output-snapshot. A data window defines which subset of events to retain. For example, a length window keeps the last N events and a time window keeps the last N seconds of events. See Chapter 14, EPL Reference: Data Windows for details.

This statement selects all Withdrawal events and instructs the runtime to remember the last five events.

select * from Withdrawal#length(5)

Upon a new Withdrawal event arriving, the runtime adds the event to the length window. It also passes the same event to callbacks.

Figure 2.6. EPL Basic Data Window Example Input and Output Flow

This statement outputs the count and total of the last five Withdrawal events.

select count(*), sum(amount) from Withdrawal#length(5)

Upon a new Withdrawal event arriving, the runtime adds the event to the length window, increases the count by one and adds the amount to the current total amount. Upon a Withdrawal event leaving the data window, the runtime decreases the count by one and subtracts its amount from the current total amount. It passes the running count and total to callbacks.

Figure 2.7. EPL Basic Data Window and Aggregation Example Input and Output Flow

The following statement outputs the count and total of the last five Withdrawal events considering only those Withdrawal events that have an amount of at least 200:

select count(*), sum(amount) from Withdrawal(amount>=200)#length(5)

Upon a new Withdrawal event arriving, and only if that Withdrawal event has an amount of 200 or more, the runtime adds the event to the length window, increases the count by one and adds the amount to the current total amount. Upon a Withdrawal event leaving the data window, the runtime decreases the count by one and subtracts its amount from the current total amount. It passes the running count and total to callbacks.

Figure 2.8. EPL Basic Filter, Data Window and Aggregation Example Input and Output Flow

For statements without a data window, the where-clause behaves the same as the filter expressions that are placed in parenthesis.

The following two statements are fully equivalent because of the absence of a data window (the .... means any select-clause expressions):

select .... from Withdrawal(amount > 200)
// equivalent to
select .... from Withdrawal where amount > 200

The next statement applies a where-clause to Withdrawal events. Where-clauses are discussed in more detail in Section 5.5, “Specifying Search Conditions: The Where Clause”.

select * from Withdrawal#length(5) where amount >= 200

The where-clause applies to both new events and expiring events. Only events that pass the where-clause are passed to callbacks.

Figure 2.9. EPL Basic Where-Clause Example Input and Output Flow

A time window is a data window that extends the specified time interval into the past. More information on time windows can be found at Section 14.3.3, “Time Window (time or win:time)”.

The next statement selects the count and total amount of Withdrawal events considering the last four seconds of events.

select count(*), sum(amount) as total from Withdrawal#time(4)

The diagram starts at a given time t and displays the contents of the time window at t + 4 and t + 5 seconds and so on.

Figure 2.10. EPL Basic Time Window Example Input and Output Flow

The statements discussed so far are not partitioned. A statement that is not partitioned implicitly has one partition. Upon deploying the un-partitioned statement the runtime allocates the single partition and it destroys the partition when your application undeploys the statement.

A partitioned statement is handy for batch processing, sessions, resetting and start/stop of your analysis. For partitioned statements you must specify a context. A context defines how partitions are allocated and destroyed. Additional information about partitioned statements and contexts can be found at Chapter 4, Context and Context Partitions.

We shall have a single partition that starts immediately and ends after four seconds:

create context Batch4Seconds start @now end after 4 sec

The next statement selects the count and total amount of Withdrawal events that arrived since the last reset (resets are at t, t+4, t+8 as so on), resetting each four seconds:

context Batch4Seconds select count(*), total(amount) from Withdrawal

Figure 2.11. EPL Basic Partitioned Statement Example Input and Output Flow

All the previous statements had continuous output. In other words, in each of previous statements output occurred as a result of a new event arriving. Use output rate limiting to output when a condition occurs, as described in Section 5.7, “Stabilizing and Controlling Output: The Output Clause”.

The next statement outputs the last count and total of all Withdrawal events every four seconds:

select count(*), total(amount) from Withdrawal output last every 4 seconds

Figure 2.12. Basic Output-Rate-Limited Statement Example Input and Output Flow

Use a partitioned statement with output rate limiting to output-and-reset. This allows you to form batches, analyze a batch and then forget all such state in respect to that batch, continuing with the next batch.

The next statement selects the count and total amount of Withdrawal events that arrived within the last four seconds at the end of four seconds, resetting after output:

create context Batch4Seconds start @now end after 4 sec

context Batch4Seconds select count(*), total(amount) from Withdrawal output last when terminated

Figure 2.13. Basic Partitioned and Output-Rate-Limited Statement Example Input and Output Flow

Named windows manage a subset of events for use by other statements. They can be selected-from, inserted- into, deleted-from and updated by multiple statements.

Tables are similar to named windows but are organized by primary keys and hold rows and columns. Tables can share aggregation state while named windows only share the subset of events they manage.

The documentation link for both is Chapter 6, EPL Reference: Named Windows and Tables. Named windows and tables can be queried with fire-and-forget queries through the API and also the inward-facing JDBC driver.

2.14.1. Named Windows

Named windows declare a window for holding events, and other statements that have the named window name in the from-clause implicitly aggregate or analyze the same set of events. This removes the need to declare the same window multiple times for different EPL statements.

The below drawing explains how named windows work.

Figure 2.14. Named Window Mechanics

The step #1 creates a named window like so:

create window WithdrawalWindow#time(10) as Withdrawal

The name of the named window is WithdrawalWindow and it will be holding the last 10 seconds of Withdrawal events (#time(10) as Withdrawal).

As a result of step #1 the runtime allocates a named window to hold 10 seconds of Withdrawal events. In the drawing the named window is filled with some events. Normally a named window starts out as an empty window however it looks nicer with some boxes already inside.

The step #2 creates an EPL statement to insert into the named window:

on Withdrawal merge WithdrawalWindow insert select *

This tells the runtime that on arrival of a Withdrawal event it must merge with the WithdrawalWindow and insert the event. The runtime now waits for Withdrawal events to arrive.

The step #3 creates an EPL statement that computes the average withdrawal amount of the subset of events as controlled by the named window:

select avg(amount) as avgAmount from WithdrawalWindow

As a result of step #3 the runtime allocates state to keep a current average. The state consists of a count field and a sum field to compute a running average. It determines that the named window is currently empty and sets the count to zero and the sum to null (if the named window was filled already it would determine the count and sum by iterating). Internally, it also registers a consumer callback with the named window to receive inserted and removed events (the insert and remove stream). The callbacks are shown in the drawing as a dotted line.

In step #4 assume a Withdrawal event arrives that has an account number of 0001 and an amount of 5000. The runtime executes the on Withdrawal merge WithdrawalWindow insert select * and thus adds the event to the time window. The runtime invokes the insert stream callback for all consumers (dotted line, internally managed callback). The consumer that computes the average amount receives the callback and the newly-inserted event. It increases the count field by one and increases the sum field by 5000. The output of the statement is avgAmount as 5000.

In step #5, which occurs 10 seconds after step #4, the Withdrawal event for account 0001 and amount 5000 leaves the time window. The runtime invokes the remove stream callback for all consumers (dotted line, internally managed callback). The consumer that computes the average amount receives the callback and the newly-removed event. It decreases the count field by one and sets the sum to null and the count is zero. The output of the statement is avgAmount as null.

2.14.2. Tables

Tables in EPL are not just holders of values of some type. EPL tables are also holders for aggregation state. Aggregations in EPL can be simple aggregations, such as count or average or standard deviation, but can also be much richer aggregations. Examples of richer aggregations are list of events (window and sorted aggregation) or a count-min-sketch (a set of hash tables that store approximations). Your application can easily extend and provide its own aggregations.

As table columns can serve as holders for aggregation state, they are a central place for updating and accessing aggregation state to be shared between statements. The below drawing explains how tables work with aggregation state.

Figure 2.15. Table Into-Table Aggregation

The step #1 creates a table like so:

create table AccountAverages(account string primary key, avgAmount avg(double))

The table that has string-typed account number as the primary key. The table also has a column that contains an average of double-type values. Note how create table does not need to know how the average gets updated, it only needs to know that the average is an average of double-type values.

As a result of step #1 the runtime allocates a table. In the drawing the table is filled with two rows for two different account numbers 0001 and 0002. Normally a table starts out as an empty table but let's assume the table has rows already.

In order to store an average of double-type values, the runtime must keep a count and a sum. Therefore in the avgAmount column of the table there are fields for count and sum. For account 0001 let's say there are currently no values and the count is zero and the sum is null.

The step #2 creates an EPL statement that aggregates the last 10 seconds of Withdrawal events:

into table AccountAverages
select avg(amount) as avgAmount
from Withdrawal#time(10)
group by account

The into table tells the compiler to store aggregations not locally as part of the statement, but into the AccountAverages table instead. The as avgAmount tells the compiler to use the column avgAmount in the table. The compiler checks that aggregation type and value types match with the table column, and that the group by-clause matches the table primary key.

The runtime looks for Withdrawal events and keeps a 10-second time window. Normally a time window starts out as an empty time window but the drawing shows a few events in the time window.

In step #3 assume a Withdrawal event arrives that has an account number of 0001 and an amount of 5000. The runtime adds the event to the time window. The runtime updates the avgAmount column of the table specifically the two fields count and sum. It increases the count field by one and increases the sum field by 5000. In the case when a row for the account number does not exist, the runtime allocates a table row and its columns and aggregation fields.

In step #4, which occurs 10 seconds after step #3, the Withdrawal event for account 0001 and amount 5000 leaves the time window. The runtime updates the avgAmount column of the table. It decreases the count field by one and sets the sum to null as the count is zero.

Other EPL statements may access table columns by putting the table into a from-clause, or by table-access-expression, on-action statement or fire-and-forget query.

The expressions in the select-clause, the use of aggregation functions and the group-by-clause are relevant to statement design. The overview herein is especially relevant to joins, on-trigger, output-rate-limiting and batch data windows.

EPL allows each aggregation function to specify its own grouping criteria. Please find further information in Section 5.6.4, “Specifying Grouping for Each Aggregation Function”. The documentation provides output examples for statement types in Appendix A, Output Reference and Samples, and the next sections outlines each statement type.

The examples below assume BankInformationWindow is a named window defined elsewhere. The examples use a join to illustrate. Joins are further described in Section 5.12, “Joining Event Streams”.

select * from Withdrawal unidirectional, BankInformationWindow

select sum(amount) 
from Withdrawal unidirectional, BankInformationWindow

select account, sum(amount) 
from Withdrawal unidirectional, BankInformationWindow

select account, sum(amount) 
from Withdrawal unidirectional, BankInformationWindow
group by account

select account, accountName, sum(amount) 
from Withdrawal unidirectional, BankInformationWindow
group by account

EPL offers the standardized match-recognize syntax for finding patterns among events. A match-recognize pattern is very similar to a regular-expression pattern.

The below statement is a sample match-recognize pattern. It detects a pattern that may be present in the events held by the named window as declared above. It looks for two immediately-followed events, i.e. with no events in-between for the same origin. The first of the two events must have high priority and the second of the two events must have medium priority.

select * from AlertNamedWindow
  match_recognize (
    partition by origin
    measures a1.origin as origin, a1.alarmNumber as alarmNumber1, a2.alarmNumber as alarmNumber2
    pattern (a1 a2)
    define
      a1 as a1.priority = 'high',
      a2 as a2.priority = 'medium'
)

The EPL pattern language is a versatile and expressive syntax for finding time and property relationships between events of many streams.

Event patterns match when an event or multiple events occur that match the pattern's definition, in a bottom-up fashion. Pattern expressions can consist of filter expressions combined with pattern operators. Expressions can contain further nested pattern expressions by including the nested expression(s) in parenthesis.

There are five types of operators:

A sample pattern that alerts on each IBM stock tick with a price greater than 80 and within the next 60 seconds:

every StockTickEvent(symbol="IBM", price>80) where timer:within(60 seconds)

A sample pattern that alerts every five minutes past the hour:

every timer:at(5, *, *, *, *)

A sample pattern that alerts when event A occurs, followed by either event B or event C:

A -> ( B or C)

A pattern where a property of a following event must match a property from the first event:

every a=EventX -> every b=EventY(objectID=a.objectID)

EPL and SQL use a three-valued logic: besides true and false, the result of expressions can also be unknown. Three-valued logic is a consequence of supporting null to mark absent data. If a null value affects the result of a logical expression, the result is neither true nor false but unknown.

A null value basically means “could be anything”. It is therefore impossible to tell whether a comparison to null is true or false. That’s where the third logical value, unknown, comes in. Unknown means “true or false, depending on the null values”.

The result of each of the following comparisons is therefore null (unknown):

null = 1

null <> 1

null > 1

null = null

Nothing equals null. Not even null equals null because each null could be different. That’s why EPL has the is null predicate to test whether a value is null or not.

Credit: Above is an excerpt from an online text that can be found at http://modern-sql.com/concept/three-valued-logic.

In EPL, an event can be represented by any of the following underlying Java objects (NEsper .NET, see Section J.6, “.NET Event Underlying Objects”):

Sometimes an event can have properties that are itself events. EPL uses the term fragment and fragment type for such event pieces. The best example is a pattern that matches two or more events and the output event contains the matching events as fragments. In other words, output events can be a composite event that consists of further events, the fragments.

Fragments have the same metadata available as their enclosing composite events. The metadata for enclosing composite events contains information about which properties are fragments, or have a property value that can be represented as a fragment and therefore as an event itself.

Fragments and type metadata can allow your application to navigate composite events without the need for using the Java reflection API and reducing the coupling to the underlying event representation. The API is further described in Section 16.5.5, “Event and Event Type”.

More information on event representations can be found in the appendix. The links are:

sendEventBean(Object event, String eventTypeName)

sendEventMap(Map map, String mapEventTypeName)

sendEventObjectArray(Object[] objectarray, String objectArrayEventTypeName)

sendEventJson(String json, String jsonEventTypeName)

sendEventAvro(Object avroGenericDataDotRecord, String avroEventTypeName)

sendEventXMLDOM(org.w3c.dom.Node node, String eventTypeName)

N/A

@EventRepresentation(map)

@EventRepresentation(objectarray)

@EventRepresentation(json)

@EventRepresentation(avro)

N/A

create schema name as class_name

create map schema name as (...)

create objectarray schema name as (...)

create json schema name as (...)

create avro schema name as (...)

@XMLSchema(...) create xml schema name as ()

create map schema ParkingEvent as (carId string, driverName string)

insert into ParkingEvent select carId, 'jim' as driverName from CarArrivalEvent

EPL does not require a fixed tuple structure and fully supports generic tuples. Event properties can be defined, added to existing types and queried at runtime.

The facilities for support of generic tuples are:

There is no need to explicitly create an event type for each tuple type. It is not necessary to create classes for tuple types at all. Events can be arbitrary objects.

The compiler uses the type information that is available: the compiler can verifies your statement against the known properties and types, preventing you as the EPL designer from making mistakes in EPL design. The compiler does not verify dynamic properties, which may return null at runtime. If type information is not available then properties are assumed to return java.lang.Object-typed values.

For example, let's say you need a generic tuple and you have Map events:

create schema GenericTuple()

Create statements that use dynamic properties, as the next EPL shows, which casts the timestamp value to a long-type value and outputs the hour-minute-second string:

select cast(timestamp?, long).format('hh mm ss') from GenericTuple

Send events like this:

Map<String, Object> genericEvent = new HashMap<>();
genericEvent.put("timestamp", new Date().getTime());
genericEvent.put("some_other_property", "hello");
runtime.getEventService().sendEventMap(genericEvent, "GenericTuple");

The first means to handle updating events is the update istream clause as further described in Section 5.20, “Updating an Insert Stream: The Update IStream Clause”. It is useful when you need to update events as they enter a stream, before events are evaluated by any particular consuming statement to that stream.

The second means to update events is the on-merge and on-update clauses, for use with tables and named windows only, as further described in Section 6.8, “Triggered Upsert Using the On-Merge Clause” and Section 6.6, “Updating Data: The On Update Clause”. On-merge is similar to the SQL merge clause and provides what is known as an "Upsert" operation: Update existing events or if no existing event(s) are found then insert a new event, all in one atomic operation provided by a single statement. On-update can be used to update individual properties of rows held in a table or named window.

To extract information from a coarse-grained event or to perform bulk operations on the rows of the property graph in an event, EPL provides a convenient syntax: When specifying a filter expression in a pattern or in a select clause, it may contain an contained-event selection syntax, as further described in Section 5.19, “Contained-Event Selection”.

For NEsper .NET also see Section J.10, “.NET Event Objects Instantiated and Populated by Insert Into”.

The insert into clause can instantiate and populate new instances of Java object events, java.util.Map events and Object[] (object array) events directly from the results of select clause expressions. Simply use the event type name as the stream name in the insert into clause as described in Section 5.10, “Merging Streams and Continuous Insertion: The Insert Into Clause”.

If instead you have an existing instance of a Java object returned by an expression, such as a single-row function or static method invocation for example, you can transpose that expression result object to a stream. This is described further in Section 5.10.7, “Transposing an Expression Result” and Section 10.4, “Select-Clause Transpose Function”.

The column names specified in the select and insert into clause must match available writable properties in the event object to be populated (the target event type). The expression result types of any expressions in the select clause must also be compatible with the property types of the target event type.

If populating a POJO-based event type and the class provides a matching constructor, the expression result types of expressions in the select clause must be compatible with the constructor parameters in the order listed by the constructor. The insert into clause column names are not relevant in this case.

Consider the following example statement:

insert into com.mycompany.NewEmployeeEvent 
select fname as firstName, lname as lastName from HRSystemEvent

The above example specifies the fully-qualified class name of NewEmployeeEvent. The runtime instantianes NewEmployeeEvent for each result row and populates the firstName and lastName properties of each instance from the result of select clause expressions. The HRSystemEvent in the example is assumed to have lname and fname properties, and either setter-methods and a default constructor, or a matching constructor.

Note how the example uses the as-keyword to assign column names that match the property names of the NewEmployeeEvent target event. If the property names of the source and target events are the same, the as-keyword is not required.

The next example is an alternate form and specifies property names within the insert into clause instead. The example also assumes that NewEmployeeEvent has been defined or imported via configuration since it does not specify the event class package name:

insert into NewEmployeeEvent(firstName, lastName) 
select fname, lname from HRSystemEvent

Finally, this example populates HRSystemEvent events. The example populates the value of a type property where the event has the value 'NEW' and populates a new event object with the value 'HIRED', copying the fname and lname property values to the new event object:

insert into HRSystemEvent 
select fname, lname, 'HIRED' as type from HRSystemEvent(type='NEW')

The matching of the select or insert into-clause column names to target event type's property names is case-sensitive. You can specify a subset of all available columns in the target event type. Wildcard (*) is allowed and copies all fields of the events or multiple events in a join.

For Java object events, your event class must provide setter-methods according to JavaBean conventions or, alternatively, a matching constructor. If the event class provides setter methods the class should also provide a default constructor taking no parameters. If the event class provides a matching constructor there is no need for setter-methods. If your event class does not have a default constructor and setter methods, or a matching constructor, your application may configure a factory method via ConfigurationEventTypeLegacy. If your event class does not have a default constructor and there is no factory method provided, the runtime uses in connection with the Oracle JVM the sun.reflect.ReflectionFactory, noting that in this case member variables do not get initialized to assigned defaults.

The compiler follows Java standards in terms of widening, performing widening automatically in cases where widening type conversion is allowed without loss of precision, for both boxed and primitive types and including BigInteger and BigDecimal.

When inserting array-typed properties into a Java, Map-type, Object-array or JSON underlying event the event definition should declare the target property as an array (Avro uses Collections).

Please note the following limitations:

Event type names of preconfigured event types are unique within both compiler and runtime.

Event type names of event types allocated by create schema are unique within both the compiler and the runtime by the combination of the deployment id and the event type name.

At runtime for internal lookups the runtime computes a CRC32 value pair. Usually you don't need to worry about CRC32 values as a collision is very unlikely and the compiler and runtime indicate relevant collisions by throwing a compile-time or deployment-time exception. For preconfigured event types the CRC32 of the type name must be unique within the runtime. For event types allocated by create schema the combination of CRC32 of the deployment id and the CRC32 of the event type name must be unique within the runtime.

This section discusses the notion of context and its role in the event processing language (EPL).

When you look up the word context in a dictionary, you may find: Context is the set of circumstances or facts that surround a particular event, situation, etc.

Context-dependent event processing occurs frequently: For example, consider a requirement that monitors banking transactions. For different customers your analysis considers customer-specific aggregations, patterns or data windows. In this example the context of detection is the customer. For a given customer you may want to analyze the banking transactions of that customer by using aggregations, data windows, patterns including other EPL constructs.

In a second example, consider traffic monitoring to detect speed violations. Assume the speed limit must be enforced only between 9 am and 5 pm. The context of detection is of temporal nature.

A context takes a cloud of events and classifies them into one or more sets. These sets are called context partitions. An event processing operation that is associated with a context operates on each of these context partitions independently. (Credit: Taken from the book "Event Processing in Action" by Opher Etzion and Peter Niblett.) A basic partitioned statement was reviewed in Section 2.11, “Basic Partitioned Statement”.

A context is a declaration of dimension and may thus result in one or more context partitions. In the banking transaction example the context dimension is the customer and a context partition exists per customer. In the traffic monitoring example there is a single context partition that exists only between 9 am and 5 pm and does not exist outside of that daily time period.

In an event processing glossary you may find the term event processing agent. An statement is an event processing agent. An alternative term for context partition is event processing agent instance.

EPL allows you to declare contexts explicitly, offering the following benefits:

EPL allows you to declare a context explicitly via the create context syntax introduced below.

After you have declared a context, one or more statements can refer to that context by specifying context name. When a statement refers to a context, all EPL-statement related state such as aggregations, patterns or data windows etc. exists once per context partition.

If a statement does not declare a context, it implicitly has a single context partition. The single context partition lives as long as the statement is deployed and ends when the statement is undeployed.

You may have heard of the term session. A context partition is the same as a session.

You may have heard of the term session window to describe the duration between when a session becomes alive to when a session gets destroyed. We use the term context partition lifecycle instead.

The context declaration specifies how the runtime manages context partitions (or sessions):

For the API to administrate context partitions please see Section 16.17, “Context Partition Administration”. For more information on locking and threading please see Section 16.8, “Runtime Threading and Concurrency”. For performance related information please refer to Chapter 23, Performance.

The create context statement declares a context by specifying a context name and context dimension information.

A context declaration by itself does not consume any resources or perform any logic until your application starts at least one statement that refers to that context. Until then the context is inactive and not in use.

When your application deploys the first statement that refers to the context, the runtime activates the context.

As soon as your application undeploys all statements that refer to the context, the context becomes inactive again.

When your application undeploys a statement that refers to a context, the context partitions associated to that statement also end (context partitions associated to other deployed statements live on).

In order for your application to undeploy the statement that declared the context it must also undeploy any statements that refer to the context.

When your application undeploys all statements that refer to that context and undeploys the statement that declared the context then the runtime removes the context declaration entirely.

The create context statement posts no output events to listeners or subscribers and does not return any rows when iterated.

create context context_name partition [by] partition_def
  [initiated [by] initiated_def] 
  [terminated [by] terminating_condition]

create context context_name partition [by]
  event_property [and event_property [and ...]] from stream_def [ [as] stream_name]
  [, event_property [...] from stream_def]   [, ...]

create context SegmentedByCustomer partition by custId from BankTxn

context SegmentedByCustomer
select custId, account, sum(amount) from BankTxn group by account

context SegmentedByCustomer
select * from pattern [
  every a=BankTxn(amount>400) -> b=BankTxn(amount>400) where timer:within(10 minutes)
]

// Neither LoginEvent nor LogoutEvent are listed in the context declaration
context SegmentedByCustomer
select * from pattern [every a=LoginEvent -> b=LogoutEvent where timer:within(10 minutes)]

initiated by stream_def [ [as] stream_name]
  [, stream_def [ [as] stream_name]]   [, ...]

create context PerCustId_TriggeredByLargeAmount
  partition by custId from BankTxn 
  initiated by BankTxn(amount>100) as largeTxn

context PerCustId_TriggeredByLargeAmount select context.largeTxn, custId, sum(amount) from BankTxn

terminated [by] terminating_condition

create context PerCustId_UntilExpired
  partition by custId from BankTxn 
  terminated by BankTxn(expired=true)

context PerCustId_UntilExpired select custId, sum(amount) from BankTxn output last when terminated

create context PerCustId_TriggeredByLargeAmount_UntilExpired
  partition by custId from BankTxn 
  initiated by BankTxn(amount>100) as txn
  terminated by BankTxn(expired=true and user=txn.user)

// Not valid
create context SegmentedByCustomer partition by custId from BankTxn, account from BankTxn

// Invalid: Type mismatch between properties
create context SegmentedByCustomer partition by custId from BankTxn, loginTime from LoginEvent

create context SegmentedByCustomer partition by 
  custId from BankTxn, loginId from LoginEvent, loginId from LogoutEvent

create context PerCust_AmountGreater100
  partition by custId from BankTxn(amount>100)
  initiated by BankTxn

context PerCust_AmountGreater100 select custId, sum(amount) from BankTxn

create context PerCust_TriggeredByLargeTxn
  partition by custId from BankTxn
  initiated by BankTxn(amount>100)

context PerCust_TriggeredByLargeTxn select custId, sum(amount) from BankTxn

create context ByCustomerAndAccount partition by custId and account from BankTxn

context ByCustomerAndAccount select custId, account, sum(amount) from BankTxn

create context ByCustomerAndAccount partition by custId and account from BankTxn

context ByCustomerAndAccount
  select context.name, context.id, context.key1, context.key2 from BankTxn

create context ByCust partition by custId from BankTxn

context ByCust 
  select * from BankTxn as t1 unidirectional, BankTxn#time(30) t2 
  where t1.amount = t2.amount

context ByCust
  select * from SecurityEvent as t1 unidirectional, BankTxn#time(30) t2 
  where t1.customerName = t2.customerName

create context context_name coalesce [by] 
  hash_func_name(hash_func_param) from stream_def
  [, hash_func_name(hash_func_param) from stream_def ]
  [, ...]
  granularity granularity_value
  [preallocate]

create context SegmentedByCustomerHash
  coalesce by consistent_hash_crc32(custId) from BankTxn granularity 16 preallocate

context SegmentedByCustomerHash
select custId, account, sum(amount) from BankTxn group by custId, account

create context MyHashContext
  coalesce by computeHash(*) from BankTxn granularity 16 preallocate

// Neither LoginEvent nor LogoutEvent are listed in the context declaration
context SegmentedByCustomerHash
select * from pattern [every a=LoginEvent -> b=LogoutEvent where timer:within(10 minutes)]

create context HashedByCustomer as coalesce
  consistent_hash_crc32(custId) from BankTxn,
  consistent_hash_crc32(loginId) from LoginEvent, 
  consistent_hash_crc32(loginId) from LogoutEvent
  granularity 32 preallocate

create context HashedByCustomer 
  coalesce consistent_hash_crc32(loginId) from LoginEvent(failed = false) 
  granularity 1024 preallocate

create context ByCustomerHash coalesce consistent_hash_crc32(custId) from BankTxn granularity 1024

context ByCustomerHash
  select context.name, context.id from BankTxn

create context context_name
  group [by] group_expression as category_label
  [, group [by] group_expression as category_label]
  [, ...]
  from stream_def

create context CategoryByTemp
  group temp < 65 as cold,
  group temp between 65 and 85 as normal,
  group temp > 85 as large
  from SensorEvent

context CategoryByTemp select context.label, count(*) from SensorEvent

context CategoryByTemp
  select context.name, context.id, context.label from SensorEvent

create context context_name
  start (@now | start_condition) 
  [ end end_condition ]

create context NineToFive start (0, 9, *, *, *) end (0, 17, *, *, *)

context NineToFive select * from TrafficEvent(speed >= 100)

create context PowerOutage start PowerOutageEvent end pattern [PowerOnEvent -> timer:interval(5)]

context PowerOutage select * from TemperatureEvent

create context Every15Minutes start @now end after 15 minutes

create context AlwaysOn start @now

create context MyCtx start MyStartEvent end MyEndEvent

context MyCtx select count(*) as cnt from MyEndEvent output when terminated

create context context_name
  initiated [by] [distinct (distinct_value_expr [,...])] [@now and] initiating_condition
  [ terminated [by] terminating_condition ]

create context CtxTrainEnter
  initiated by TrainEnterEvent as te
  terminated after 5 minutes

context CtxTrainEnter
select t1 from pattern [
  t1=TrainEnterEvent(trainId = context.te.trainId) -> timer:interval(5 min) and not TrainLeaveEvent(trainId = context.te.trainId)
  ]

create context CtxEachMinute
  initiated @now and pattern [every timer:interval(1 minute)]
  terminated after 1 minutes

context CtxEachMinute select avg(temp) from SensorEvent

create context CtxTrainEnter initiated by TrainEnterEvent as te

create context OrderContext
  initiated by distinct(orderId) NewOrderEvent as newOrder 
  terminated by CloseOrderEvent(closeOrderId = newOrder.orderId)

context NineToFive 
select context.name, context.startTime, context.endTime from TrafficEvent(speed >= 100)

context CtxTrainEnter
select *, context.te.trainId, context.id, context.name
from TrainLeaveEvent(trainId = context.te.trainId)

event_stream_name [(filter_criteria)] [as stream_name]

// A non-overlapping context that starts when MyStartEvent arrives and ends when MyEndEvent arrives
create context MyContext start MyStartEvent end MyEndEvent

// An overlapping context where each MyEvent with level greater zero 
// initiates a new context partition that terminates after 10 seconds
create context MyContext initiated MyEvent(level > 0) terminated after 10 seconds

// A non-overlapping context that starts when MyEvent arrives
// and ends when a matching MyEvent arrives (same id)
create context MyContext 
start MyEvent as myevent
end MyEvent(id=myevent.id)

// An overlapping context where each MyInitEvent initiates a new context partition 
// that terminates when a matching MyTermEvent arrives 
create context MyContext 
initiated by MyInitEvent as e1 
terminated by MyTermEvent(id=e1.id, level <> e1.level)

pattern [pattern_expression] [@inclusive]

// A non-overlapping context that starts when either StartEventOne or StartEventTwo arrive
// and that ends after 5 seconds.
// Here neither StartEventOne or StartEventTwo count towards any statements
// that are referring to the context.
create context MyContext 
  start pattern [StartEventOne or StartEventTwo] 
  end after 5 seconds

// Same as above. 
// Here both StartEventOne or StartEventTwo do count towards any statements
// that are referring to the context.
create context MyContext 
  start pattern [a=StartEventOne or b=StartEventTwo] @inclusive 
  end after 5 seconds

// An overlapping context where each distinct MyInitEvent initiates a new context 
// and each context partition terminates after 20 seconds
// Use @inclusive to say that the same MyInitEvent that fires the pattern
// also applies to statements that are associated to the context.
create context MyContext
  initiated by pattern [every-distinct(a.id, 20 sec) a=MyInitEvent]@inclusive
  terminated after 20 sec

// An overlapping context where each pattern match initiates a new context 
// and all context partitions terminate when MyTermEvent arrives.
// The MyInitEvent and MyOtherEvent that trigger the pattern are themselves not included 
// in any statements that are associated to the context.
create context MyContext
  initiated by pattern [every MyInitEvent -> MyOtherEvent where timer:within(5)]
  terminated by MyTermEvent

// A non-overlapping context that starts when either StartEventOne or StartEventTwo arrive
// and that ends when either a matching EndEventOne or EndEventTwo arrive
create context MyContext 
  start pattern [a=StartEventOne or b=StartEventTwo]@inclusive
  end pattern [EndEventOne(id=a.id) or EndEventTwo(id=b.id)]

// A non-overlapping context started daily between 9 am to 5 pm
// and not started outside of these hours:
create context NineToFive start (0, 9, *, *, *) end (0, 17, *, *, *)

// An overlapping context where crontab initiates a new context every 1 minute
// and each context partition terminates after 10 seconds:
create context MyContext initiated (*, *, *, *, *) terminated after 10 seconds

// From  8:00am (see start) to 9am (see end) and from 10am (see start) to 12pm (see end).
create context MyContext
  start (0, 8, *, *, *, *), (0, 10, *, *, *, *)
  end (0, 9, *, *, *, *), (0, 12, *, *, *, *)

// Monday to Friday from 8am to 11am, Saturday and Sunday from 8am to 8pm.
create context MyContext
  start (0, 8, *, *, *, *)
  end (0, 11, *, *, [1, 2, 3, 4, 5], *), (0, 20, *, *, [0, 6], *)

// Monday from 8am to 10am, Tuesday from 9am to 10am
create context MyContext
  start (0, 8, *, *, 1, *), (0, 9, *, *, 2, *)
  end (0, 10, *, *, *)

after time_period_expression

// A non-overlapping context started after 10 seconds 
// that ends 1 minute after it starts and that again starts 10 seconds thereafter.
create context NonOverlap10SecFor1Min start after 10 seconds end after 1 minute

// An overlapping context that starts a new context partition every 5 seconds
// and each context partition lasts 1 minute
create context Overlap5SecFor1Min initiated after 5 seconds terminated after 1 minute

A nested context is a context that is composed from two or more contexts.

The syntax for creating a nested context is as follows:

create context context_name
  context nested_context_name [as] nested_context_definition ,
  context nested_context_name [as] nested_context_definition [, ...]

The context_name you assign to the context can be any identifier.

Following the context name is a comma-separated list of nested contexts. For each nested context specify the context keyword followed a nested context name and the nested context declaration. Any of the context declarations as outlined in Section 4.2, “Context Declaration” are allowed for nested contexts. The order of nested context declarations matters as outlined below. The nested context names have meaning only in respect to built-in properties and statements may not be assigned to nested context names.

The next statement creates a nested context NineToFiveSegmented that, between 9 am and 5 pm, allocates a new context partition for each customer id:

create context NineToFiveSegmented
  context NineToFive start (0, 9, *, *, *) end (0, 17, *, *, *),
  context SegmentedByCustomer partition by custId from BankTxn

The following statement refers to the nested context to compute a total withdrawal amount per account for each customer but only between 9 am and 5 pm:

context NineToFiveSegmented
select custId, account, sum(amount) from BankTxn group by account

EPL implements nested contexts as a context tree: The context declared first controls the lifecycle of the context(s) declared thereafter. Thereby, in the above example, outside of the 9am-to-5pm time the runtime has no memory and consumes no resources in relationship to bank transactions or customer ids.

When combining segmented contexts, the set of context partitions for the nested context effectively is the Cartesian product of the partition sets of the nested segmented contexts.

When combining temporal contexts with other contexts, since temporal contexts may overlap and may terminate, it is important to understand that temporal contexts control the lifecycle of sub-contexts (contexts declared thereafter). The order of declaration of contexts in a nested context can thereby change resource usage and output result.

The next statement creates a context that allocates context partition only when a train enters a station and then for each hash of the tag id of a passenger as indicated by PassengerScanEvent events, and terminates all context partitions after 5 minutes:

create context CtxNestedTrainEnter
  context InitCtx initiated by TrainEnterEvent as te terminated after 5 minutes,
  context HashCtx coalesce by consistent_hash_crc32(tagId) from PassengerScanEvent
    granularity 16 preallocate

In the example above the runtime does not start tracking PassengerScanEvent events or hash codes or allocate context partitions until a TrainEnterEvent arrives.

create context CtxSampleNestedContext
  context SpanA start AStart end AEnd,
  context SpanB start BStart end BEnd

context CtxSampleNestedContext select count(*) from C

create context CtxSampleNestedContext 
  start pattern[every a=AStart or every a=BStart] as mypattern 
  end pattern[every AEnd or every BEnd]

context CtxNestedTrainEnter
select context.InitCtx.te.trainId, context.HashCtx.id,
  tagId, count(*) from PassengerScanEvent group by tagId

context NineToFiveSegmented
select context.NineToFive.startTime, context.SegmentedByCustomer.key1 from BankTxn

context NineToFiveSegmented select context.name, context.id from BankTxn

You do not need to declare a context to partition data windows, aggregation values or patterns themselves individually. You may mix-and-match partitioning as needed.

The table below outlines other partitioning syntax supported by EPL:

// Length window of 2 events per customer
select * from
  BankTxn#groupwin(custId)#length(2)

select avg(price), window(*)
  from BankTxn group by custId

select * from pattern [
  every a=BankTxn -> BankTxn(custId = a.custId)...]

select * from match_recognize 
  ... partition by custId

select * from ... where a.custId = b.custId

You may use output rate limiting to trigger output when a context partition ends or terminates using output when terminated. This concept was introduced to you in Section 2.13, “Basic Partitioned and Output-Rate-Limited Statement”. For more information on output rate limiting please see Section 5.7, “Stabilizing and Controlling Output: The Output Clause”.

The context CtxEachMinute initiates a new context partition every 1 minute, and each context partition expires after 5 minutes:

create context CtxEachMinute
initiated by pattern [every timer:interval(1 min)]
terminated after 5 minutes

The following statement computes an ongoing average temperature however only outputs the last value of the average temperature, together with the context partition id and the current runtime time, after 5 minutes when a context partition ends:

context CtxEachMinute
select current_timestamp as endtime, context.id, avg(temp) from SensorEvent output last when terminated

The when terminated syntax can be combined with other output rates.

The next example outputs every 1 minute and also when the context partition ends:

context CtxEachMinute
select context.id, avg(temp) from SensorEvent output last every 1 minute and when terminated

In the case that the end/terminating condition of the context partition is an event or pattern, the context properties contain the information of the tagged events in the pattern or the single event that ended/terminated the context partition.

For example, consider the following context wherein the runtime initializes a new context partition for each arriving MyStartEvent event and that terminates a context partition when a matching MyEndEvent arrives:

create context CtxSample
initiated by MyStartEvent as startevent
terminated by MyEndEvent(id = startevent.id) as endevent

The following statement outputs the id property of the initiating and terminating event and only outputs when a context partition ends:

context CtxSample 
select context.startevent.id, context.endevent.id, count(*) from MyEvent
output last when terminated

You may in addition specify a termination expression that the runtime evaluates when a context partition terminates. Only when the termination expression evaluates to true does output occur. The expression may refer to built-in properties as described in Section 5.7.1.1, “Controlling Output Using an Expression”. The syntax is as follows:

...output when terminated and termination_expression

The next example statement outputs when a context partition ends but only if at least two events are available for output:

context CtxEachMinute
select * from SensorEvent output when terminated and count_insert >= 2

The final example EPL outputs when a context partition ends and sets the variable myvar to a new value:

context CtxEachMinute
select * from SensorEvent output when terminated then set myvar=3

Named windows are globally-visible data windows that may be referred to by multiple statements. You may refer to named windows in statements that declare a context without any special considerations, with the exception of on-action statements (latter must refer to the same context associated with the named window).

You may also create a named window and declare a context for the named window. In this case the runtime in effect manages separate named windows, one for each context partition.

For example, consider the 9 am to 5 pm non-overlapping context as shown earlier:

create context NineToFive start (0, 9, *, *, *) end (0, 17, *, *, *)

You may create a named window that only exists between 9 am and 5 pm:

context NineToFive create window SpeedingEvents1Hour#time(30 min) as TrafficEvent

You can insert into the named window:

context NineToFive on TrafficEvent(speed > 100) merge SpeedingEvents1Hour insert select *

Any on-merge, on-select, on-update and on-delete statements must declare the same context, in order to operate on partitioned named windows or tables for the same partition.

The following is not a valid statement as it does not declare the same context that was used to declare the named window:

// You must declare the same context for on-trigger statements
on TruncateEvent delete from SpeedingEvents1Hour

The following is valid:

context NineToFive on TruncateEvent delete from SpeedingEvents1Hour

For context declarations that require specifying event types, such as the hash segmented context and keyed segmented context, please provide the named window underlying event type.

The following sample statements define a type for the named window, declare a context and associate the named window to the context:

create schema ScoreCycle (userId string, keyword string, productId string, score long)

create context HashByUserCtx as 
  coalesce by consistent_hash_crc32(userId) from ScoreCycle granularity 64

context HashByUserCtx create window ScoreCycleWindow#unique(productId, keyword) as ScoreCycle

Tables are globally-visible data structures that hold rows organized by primary key(s) and that may be referred to by multiple statements. You may refer to tables in statements that declare a context without any special considerations, with the exception of on-action statements (latter must refer to the same context associated with the table).

You may also create a table and declare a context for the table. In this case the runtime in effect manages separate tables, one for each context partition.

For example, consider the 9 am to 5 pm non-overlapping context as shown earlier:

create context NineToFive start (0, 9, *, *, *) end (0, 17, *, *, *)

You may create a table that only exists between 9 am and 5 pm:

context NineToFive create table AverageSpeedTable (
  carId string primary key, 
  avgSpeed avg(double))

You can aggregate-into the table only if the aggregating statement declares the same context:

// declare the same context as for the table
context NineToFive into table AverageSpeedTable
select avg(speed) as avgSpeed
from TrafficEvent
group by carId

When you declare a context for a table, any select, on-merge, on-select, on-update and on-delete statements as well as statements that subquery the table must declare the same context.

For example, this EPL truncates the AverageSpeedTable:

context NineToFive on TruncateEvent delete from AverageSpeedTable

A variable is a scalar, object or event value that is available for use in all statements. Variables can be either global variables or context variables.

The value of a global variable is the same for all context partitions. The next example declares a global threshold variable:

create variable integer var_global_threshold = 100

For context variables, there is a variable value per context partition. The next example declares a context and a context variable:

create context ParkingLotContext initiated by CarArrivalEvent as cae terminated by CarDepartureEvent(lot = cae.lot)

context ParkingLotContext create variable integer var_parkinglot_threshold = 100

The variable var_parkinglot_threshold is a context variable. Each context partition can have its own value for the variable.

For more information on variables, please refer to Section 5.17, “Variables and Constants”.

Context variables can only be used in statements that associated to the same context.

The API to read and manage content partitions themselves is Section 16.17, “Context Partition Administration”.

Selecting specific context partitions and interrogating context partition state is useful for:

The runtime provides APIs to identify, filter and select context partitions for statement iteration and fire-and-forget queries. The APIs are described in detail at Section 16.16, “Context Partition Selection”.

For statement iteration, your application can provide context selector objects to the iterate and safeIterate methods on EPStatement. If your code does not provide context selectors the iteration considers all context partitions. At the time of iteration, the runtime obtains the current set of context partitions and iterates each independently. If your statement has an order-by clause, the order-by clause orders within the context partition and does not order across context partitions.

For fire-and-forget queries, your application can provide context selector objects to the executeQuery method on EPFireAndForgetService and to the execute method on EPFireAndForgetPreparedQuery. If your code does not provide context selectors the fire-and-forget query considers all context partitions. At the time of fire-and-forget query execution, the runtime obtains the current set of context partitions and queries each independently. If the fire-and-forget query has an order-by clause, the order-by clause orders within the context partition and does not order across context partitions.