www.espertech.comDocumentation
This section outlines the different means to model and represent events.
EPL uses the term event type to describe the type information available for an event representation.
Your application may configure predefined event types using the configuration object or dynamically add event types using create schema
.
The EPL create schema
syntax allows declaring an event type at runtime using EPL, see Section 5.15, “Declaring an Event Type: Create Schema”.
Section 16.5.5, “Event and Event Type” explains how an event type becomes visible in statements and output events delivered by the runtime.
An event is an immutable record of a past occurrence of an action or state change. Event properties capture the state information for an event.
In EPL, an event can be represented by any of the following underlying Java objects (NEsper .NET, see Section J.7, “.NET Event Underlying Objects”):
Table 3.1. Event Underlying Java Objects
Java Class | Description |
---|---|
java.lang.Object | Any Java POJO (plain-old java object) with getter methods following JavaBean conventions; Legacy Java classes not following JavaBean conventions can also serve as events . |
java.util.Map | Map events are implementations of the java.util.Map interface where each map entry is a propery value. |
Object[] (array of object) | Object-array events are arrays of objects (type Object[] ) where each array element is a property value. |
String (or JsonEventUnderlying ) | JSON events are string-typed JSON-formatted documents. The parsed JSON document is available and implements the JsonEventUnderlying interface. |
org.apache.avro.generic.GenericData.Record | Apache Avro events are GenericData.Record objects (Avro is a data serialization system with schema support) |
org.w3c.dom.Node | XML document object model (DOM). |
EPL provides multiple choices for representing an event. There is no absolute need for you to create new Java classes to represent an event.
Event representations have the following in common:
All event representations support nested, indexed and mapped properties (aka. property expression), as explained in more detail below. There is no limitation to the nesting level.
All event representations provide event type metadata. This includes type metadata for nested properties.
All event representations allow transposing the event itself and parts or all of its property graph into new events. The term transposing refers to selecting the event itself or event properties that are themselves nestable property graphs, and then querying the event's properties or nested property graphs in further statements.
The Java object, Map, Object-array, JSON and Avro representations allow supertypes.
The API behavior for all event representations is the same, with minor exceptions noted.
The benefits of multiple event representations are:
For applications that already have events in one of the supported representations, there is no need to transform events before processing for both input and output.
Event representations are exchangeable, reducing or eliminating the need to change statements when the event representation changes, i.e. the EPL does not depend on whether events are Objects, Map(s), Object-array(s), JSON document(s), Avro record(s) or XML document(s).
Event representations are interoperable, allowing all event representations to interoperate in same or different statements.
The choice makes it possible to consciously trade-off performance, ease-of-use, the ability to evolve and effort needed to import or externalize events and use existing event type metadata.
Event properties capture the state information for an event. Event properties can be simple, indexed, mapped and nested event properties.
The table below outlines the different types of properties and their syntax in an event expression:
Table 3.2. Types of Event Properties
Type | Description | Syntax | Example |
---|---|---|---|
Simple | A property that has a single value that may be retrieved. | name | sensorId |
Indexed | An indexed property stores an ordered collection of objects (all of the same type) that can be individually accessed by an integer-valued, non-negative index (or subscript). | name[index] | temperature[0] |
Mapped | A mapped property stores a keyed collection of objects (all of the same type). | name('key') | isTurnedOn('light') |
Nested | A nested property is a property that lives within another property of an event. | name.nestedname | sensor.value |
Combinations are also possible. For example, a valid combination could be person.address('home').street[0]
.
You may use any expression as a mapped property key or indexed property index by putting the expression within parenthesis after the mapped or index property name. Please find examples below.
If your application uses java.util.Map
, Object[]
(object-array) or JSON or XML to represent events, then event property names may themselves contain the dot ('.') character. The backslash ('\') character can be used to escape dot characters in property names, allowing a property name to contain dot characters.
For example, the EPL as shown below expects a property by name part1.part2
to exist on event type MyEvent
:
select part1\.part2 from MyEvent
Sometimes your event properties may overlap with EPL language keywords or contain spaces or other special characters. In this case you may use the backwards apostrophe `
(aka. back tick) character to escape the property name.
The next example assumes a Quote
event that has a property by name order
, while order
is also a reserved keyword:
select `order`, price as `price.for.goods` from Quote
When escaping mapped or indexed properties, make sure the back tick character appears outside of the map key or index.
The next EPL selects event properties that have names that contain spaces (e.g. candidate book
), have the tick special character (e.g. children's books
), are an indexed property (e.g. children's books[0]
) and a mapped property
that has a reserved keyword as part of the property name (e.g. book select('isbn')
):
select `candidate book` , `children's books`[0], `book select`('isbn') from MyEventType
When referring to nested properties, escape at the individual nesting level, for example (assuming MyEventType
has a payload
property that is itself an event):
select payload.`candidate book`, payload.`other books`? from MyEventType
Avro does not support the dot-character in field names.
The key or index expression must be placed in parenthesis. When using an expression as key for a mapped property, the expression must return a String
-typed value. When using an expression as index for an indexed property, the expression must return an int
-typed value.
This example below uses Java classes to illustrate; The same principles apply to all event representations.
Assume a class declares these properties (getters not shown for brevity):
public class MyEventType { String myMapKey; int myIndexValue; int myInnerIndexValue; Map<String, InnerType> innerTypesMap; // mapped property InnerType[] innerTypesArray; // indexed property } public class InnerType { String name; int[] ids; }
A sample statement demonstrating expressions as map keys or indexes is:
select innerTypesMap('somekey'), // returns map value for 'somekey' innerTypesMap(myMapKey), // returns map value for myMapKey value (an expression) innerTypesArray[1], // returns array value at index 1 innerTypesArray(myIndexValue) // returns array value at index myIndexValue (an expression) from MyEventType
The dot-operator can be used to access methods on the value objects returned by the mapped or indexed properties. By using the dot-operator the syntax follows the chained method invocation described at Section 9.7, “Dot Operator”.
A sample statement demonstrating the dot-operator as well as expressions as map keys or indexes is:
select innerTypesMap('somekey').ids[1], innerTypesMap(myMapKey).getIds(myIndexValue), innerTypesArray[1].ids[2], innerTypesArray(myIndexValue).getIds(myInnerIndexValue) from MyEventType
Please note the following limitations:
The square brackets-syntax for indexed properties does now allow expressions and requires a constant index value.
When using the dot-operator with mapped or indexed properties that have expressions as map keys or indexes you must follow the chained method invocation syntax.
Dynamic (unchecked) properties are event properties that need not be known at statement compilation time. Such properties are resolved during runtime: they provide duck typing functionality.
The idea behind dynamic properties is that for a given underlying event representation you don't always know all properties in advance. An underlying event may have additional properties that are not known at statement compilation time, that you want to query on. The concept is especially useful for events that represent rich, object-oriented domain models.
The syntax of dynamic properties consists of the property name and a question mark. Indexed, mapped and nested properties can also be dynamic properties:
Table 3.3. Types of Event Properties
Type | Syntax |
---|---|
Dynamic Simple | name? |
Dynamic Indexed | name[index]? |
Dynamic Mapped | name('key')? |
Dynamic Nested | name?.nestedPropertyName |
Dynamic properties always return the java.lang.Object
type. Also, dynamic properties return a null
value if the dynamic property does not exist on events processed at runtime.
As an example, consider an OrderEvent event that provides an "item" property. The "item" property is of type Object
and holds a reference to an instance of either a Service or Product.
Assume that both Service and Product classes provide a property named "price". Via a dynamic property you can specify a statement that obtains the price property from either object (Service or Product):
select item.price? from OrderEvent
As a second example, assume that the Service class contains a "serviceName" property that the Product class does not possess. The following statement returns the value of the "serviceName" property for Service objects. It returns a null
-value for Product objects that do not have the "serviceName" property:
select item.serviceName? from OrderEvent
Consider the case where OrderEvent has multiple implementation classes, some of which have a "timestamp" property. The next statement returns the timestamp property of those implementations of the OrderEvent interface that feature the property:
select timestamp? from OrderEvent
The statement as above returns a single column named "timestamp?" of type Object
.
When dynamic properties are nested, then all properties under the dynamic property are also considered dynamic properties. In the below example the statement asks for the "direction" property of the object returned by the "detail" dynamic property:
select detail?.direction from OrderEvent
Above is equivalent to:
select detail?.direction? from OrderEvent
The functions that are often useful in conjunction with dynamic properties are:
The cast
function casts the value of a dynamic property (or the value of an expression) to a given type.
The exists
function checks whether a dynamic property exists. It returns true
if the event has a property of that name, or false if the property does not exist on that event.
The instanceof
function checks whether the value of a dynamic property (or the value of an expression) is of any of the given types.
The typeof
function returns the string type name of a dynamic property.
Sometimes an event can have properties that are itself events. EPL uses the term fragment and fragment type for such event pieces. The best example is a pattern that matches two or more events and the output event contains the matching events as fragments. In other words, output events can be a composite event that consists of further events, the fragments.
Fragments have the same metadata available as their enclosing composite events. The metadata for enclosing composite events contains information about which properties are fragments, or have a property value that can be represented as a fragment and therefore as an event itself.
Fragments and type metadata can allow your application to navigate composite events without the need for using the Java reflection API and reducing the coupling to the underlying event representation. The API is further described in Section 16.5.5, “Event and Event Type”.
More information on event representations can be found in the appendix. The links are:
Table 3.4. Comparing Event Representations
Event Representation | More Information and Examples |
---|---|
Java Object (POJO/Bean or other) | Appendix D, Event Representation: Plain-Old Java Object Events |
Map | Appendix E, Event Representation: java.util.Map Events |
Object-array | Appendix F, Event Representation: Object-Array (Object[]) Events |
JSON | Appendix G, Event Representation: JSON Events |
Avro | Appendix H, Event Representation: Avro Events (org.apache.avro.generic.GenericData.Record) |
XML Document | Appendix I, Event Representation: org.w3c.dom.Node XML Events |
For sending incoming events into the runtime for processing, your application uses one of the send-event methods on the EPEventService
interface:
Table 3.5. EPEventService Send-Event Methods
Event Representation | Method for Processing Events |
---|---|
Java Object (POJO/Bean or other) | sendEventBean(Object event, String eventTypeName) |
Map | sendEventMap(Map map, String mapEventTypeName) |
Object-array | sendEventObjectArray(Object[] objectarray, String objectArrayEventTypeName) |
JSON | sendEventJson(String json, String jsonEventTypeName) |
Avro | sendEventAvro(Object avroGenericDataDotRecord, String avroEventTypeName) |
XML Document | sendEventXMLDOM(org.w3c.dom.Node node, String eventTypeName) |
Please find an example in the respective appendix.
The StatementUpdateListener
interface receives statement output. The output events can be either of the representations
Table 3.6. Annotation for Receiving Events
Event Representation | Annotation |
---|---|
Java Object (POJO/Bean or other) | N/A |
Map | @EventRepresentation(map) |
Object-array | @EventRepresentation(objectarray) |
JSON | @EventRepresentation(json) |
Avro | @EventRepresentation(avro) |
XML Document | N/A |
Please find an example in the respective appendix.
The create-schema clause can be used to define an event type and its event representation.
Table 3.7. Create-Schema
Event Representation | Annotation |
---|---|
Java Object (POJO/Bean or other) | create schema name as class_name |
Map | create map schema name as (...) |
Object-array | create objectarray schema name as (...) |
JSON | create json schema name as (...) |
Avro | create avro schema name as (...) |
XML Document | @XMLSchema(...) create xml schema name as () |
Your statements can use create schema
and insert into
to define an event type and to produce events of the type.
In the following example the first statement declares a schema and the second statement inserts events according to the schema:
create map schema ParkingEvent as (carId string, driverName string)
insert into ParkingEvent select carId, 'jim' as driverName from CarArrivalEvent
Please find additional examples in Section 5.15, “Declaring an Event Type: Create Schema”.
Each of the event representations of Java object, Map, Object-array, JSON, Avro and XML document has advantages and disadvantages that are summarized in the table below:
Table 3.8. Comparing Event Representations
Java Object (POJO/Bean or other) | Map | Object-array | JSON | Avro | XML Document | |
---|---|---|---|---|---|---|
Performance | Excellent | Good | Excellent | Excellent | Very Good | Not comparable and depending on use of XPath |
Memory Use | Small | Medium | Small | Small | Small | Depends on DOM and XPath implementation used, can be large |
Call Method on Event | Yes | Yes, if contains Object(s) | Yes, if contains Object(s) | No | No | No |
Nested, Indexed, Mapped and Dynamic Properties | Yes | Yes | Yes | Yes | Yes | Yes |
Course-grained event syntax | Yes | Yes | Yes | Yes | Yes | Yes |
Insert-into that Representation | Yes | Yes | Yes | Yes | Yes | No |
Create-schema Syntax | Yes | Yes | Yes | Yes | Yes | Yes |
Object is Self-Descriptive | Yes | Yes | No | Yes | Yes | Yes |
Supertypes | Multiple | Multiple | Single | Single | Single | No |
EPL does not require a fixed tuple structure and fully supports generic tuples. Event properties can be defined, added to existing types and queried at runtime.
The facilities for support of generic tuples are:
Dynamic properties allow querying properties that are not defined, see Section 3.3, “Dynamic Event Properties”.
The cast
function for operations that require strongly-typed data, see Section 10.1.2, “The Cast Function”.
Type inheritance for adding properties to supertypes, see Section 5.15, “Declaring an Event Type: Create Schema”
The Map event representation, as it allows any map key to become an event property, see Appendix E, Event Representation: java.util.Map Events
The JSON event representation, as it allows any JSON value (JSON objects, arrays, all JSON types) to become an event property, see Appendix G, Event Representation: JSON Events
The Avro event representation, as it allows any Avro field to become an event property, see Appendix H, Event Representation: Avro Events (org.apache.avro.generic.GenericData.Record)
The POJO event representation, as getter-methods and fields can be dynamically discovered to become an event property, see Appendix D, Event Representation: Plain-Old Java Object Events
The XML event representation, as the DOM can have any attribute or nested element and there does not need to be a schema, see Appendix I, Event Representation: org.w3c.dom.Node XML Events
Event types can be deployed and undeploy at runtime using create schema
and the deployment API
There is no need to explicitly create an event type for each tuple type. It is not necessary to create classes for tuple types at all. Events can be arbitrary objects.
The compiler uses the type information that is available: the compiler can verifies your statement against the known properties and types,
preventing you as the EPL designer from making mistakes in EPL design.
The compiler does not verify dynamic properties, which may return null
at runtime.
If type information is not available then properties are assumed to return java.lang.Object
-typed values.
For example, let's say you need a generic tuple and you have Map
events:
create schema GenericTuple()
Create statements that use dynamic properties, as the next EPL shows, which casts the timestamp value to a long
-type value and outputs the hour-minute-second string:
select cast(timestamp?, long).format('hh mm ss') from GenericTuple
Send events like this:
Map<String, Object> genericEvent = new HashMap<>(); genericEvent.put("timestamp", new Date().getTime()); genericEvent.put("some_other_property", "hello"); runtime.getEventService().sendEventMap(genericEvent, "GenericTuple");
To summarize, an event is an immutable record of a past occurrence of an action or state change, and event properties contain useful information about an event.
The length of time an event is of interest to the runtime (retention time) depends on your statements, and especially the data window, pattern and output rate limiting clauses of your statements.
During the retention time of an event more information about the event may become available, such as additional properties or changes to existing properties. EPL provides two concepts for handling updates to events.
The first means to handle updating events is the update istream
clause as further described in Section 5.20, “Updating an Insert Stream: The Update IStream Clause”. It is useful when you need to update events as they enter a stream, before events are evaluated by any particular
consuming statement to that stream.
The second means to update events is the on-merge
and on-update
clauses, for use with tables and named windows only, as further described in Section 6.8, “Triggered Upsert Using the On-Merge Clause” and Section 6.6, “Updating Data: The On Update Clause”. On-merge
is similar to the SQL merge
clause and provides what is known as an "Upsert" operation: Update existing events or if no existing event(s) are found then insert a new event, all in one atomic operation provided by a single statement.
On-update can be used to update individual properties of rows held in a table or named window.
Your application events may consist of fairly comprehensive, coarse-grained structures or documents. For example in business-to-business integration scenarios, XML documents or other event objects can be rich deeply-nested graphs of event properties.
To extract information from a coarse-grained event or to perform bulk operations on the rows of the property graph in an event, EPL provides a convenient syntax:
When specifying a filter expression in a pattern or in a select
clause, it may contain an contained-event selection syntax, as further described in Section 5.19, “Contained-Event Selection”.
For NEsper .NET also see Section J.11, “.NET Event Objects Instantiated and Populated by Insert Into”.
The insert into
clause can instantiate and populate new instances of Java object events, java.util.Map
events and Object[]
(object array) events directly from the results of select
clause expressions. Simply use the event type name as the stream name in the insert into
clause as described in Section 5.10, “Merging Streams and Continuous Insertion: The Insert Into Clause”.
If instead you have an existing instance of a Java object returned by an expression, such as a single-row function or static method invocation for example, you can transpose that expression result object to a stream. This is described further in Section 5.10.7, “Transposing an Expression Result” and Section 10.4, “Select-Clause Transpose Function”.
The column names specified in the select
and insert into
clause must match available writable properties in the event object to be populated (the target event type). The expression result types of any expressions in the select
clause must also be compatible with the property types of the target event type.
If populating a POJO-based event type and the class provides a matching constructor, the expression result types of expressions in the select
clause must be compatible with the constructor parameters in the order listed by the constructor. The insert into
clause column names are not relevant in this case.
Consider the following example statement:
insert into com.mycompany.NewEmployeeEvent select fname as firstName, lname as lastName from HRSystemEvent
The above example specifies the fully-qualified class name of NewEmployeeEvent
. The runtime instantianes NewEmployeeEvent
for each result row and populates the firstName
and lastName
properties of each instance from the result of select
clause expressions. The HRSystemEvent
in the example is assumed to have lname
and fname
properties, and either setter-methods and a default constructor, or a matching constructor.
Note how the example uses the as
-keyword to assign column names that match the property names of the NewEmployeeEvent
target event. If the property names of the source and target events are the same, the as
-keyword is not required.
The next example is an alternate form and specifies property names within the insert into
clause instead. The example also assumes that NewEmployeeEvent
has been defined or imported via configuration since it does not specify the event class package name:
insert into NewEmployeeEvent(firstName, lastName) select fname, lname from HRSystemEvent
Finally, this example populates HRSystemEvent
events. The example populates the value of a type
property where the event has the value 'NEW' and populates a new event object with the value 'HIRED', copying the fname
and lname
property values to the new event object:
insert into HRSystemEvent select fname, lname, 'HIRED' as type from HRSystemEvent(type='NEW')
The matching of the select
or insert into
-clause column names to target event type's property names is case-sensitive. You can specify a subset of all available columns in the target event type. Wildcard (*
) is allowed and copies all fields of the events or multiple events in a join.
For Java object events, your event class must provide setter-methods according to JavaBean conventions or, alternatively, a matching constructor. If the event class provides setter methods the class should also provide a default constructor taking no parameters. If the event class provides a matching constructor there is no need for setter-methods. If your event class does not have a default constructor and setter methods, or a matching constructor, your application may configure a factory method via ConfigurationEventTypeLegacy
. If your event class does not have a default constructor and there is no factory method provided, the runtime uses in connection with the Oracle JVM the sun.reflect.ReflectionFactory
, noting that in this case member variables do not get initialized to assigned defaults.
The compiler follows Java standards in terms of widening, performing widening automatically in cases where widening type conversion is allowed without loss of precision, for both boxed and primitive types and including BigInteger and BigDecimal.
When inserting array-typed properties into a Java, Map-type, Object-array or JSON underlying event the event definition should declare the target property as an array (Avro uses Collections).
Please note the following limitations:
Event types that utilize XML org.w3c.dom.Node
underlying event objects cannot be target of an insert into
clause.
Event type names of preconfigured event types are unique within both compiler and runtime.
Event type names of event types allocated by create schema
are unique within both the compiler and the runtime by the combination of the deployment id and the event type name.
At runtime for internal lookups the runtime computes a CRC32 value pair. Usually you don't need to worry about CRC32 values as a collision is very unlikely and the compiler and runtime indicate relevant collisions
by throwing a compile-time or deployment-time exception.
For preconfigured event types the CRC32 of the type name must be unique within the runtime.
For event types allocated by create schema
the combination of CRC32 of the deployment id and the CRC32 of the event type name must be unique within the runtime.