Chapter 2. Event Representations

This section outlines the different means to model and represent events.

Esper uses the term event type to describe the type information available for an event representation.

Your application may configure predefined event types at startup time or dynamically add event types at runtime via API or EPL syntax. See Section 13.4, “Configuration Items” for startup-time configuration and Section 12.3.7, “Runtime Configuration” for the runtime configuration API.

The EPL create schema syntax allows declaring an event type at runtime using EPL, see Section 4.16, “Declaring an Event Type: Create Schema”.

In Section 12.5, “Event and Event Type” we explain how an event type becomes visible in EPL statements and output events delivered by the engine.

2.1. Event Underlying Objects

An event is an immutable record of a past occurrence of an action or state change. Event properties capture the state information for an event.

In Esper, an event can be represented by any of the following underlying objects:

Table 2.1. Event Underlying Vanilla Objects

CLR TypeDescription
System.ObjectAny CLR type with read accessible properties.
System.Collection.Generic.IDictionary<string, object>Map (dictionary) events are key-values pairs and can also contain objects, further Map, and arrays thereof.
System.Xml.XmlNodeXML document object model (DOM).
Application classesPlug-in event representation via the extension API.

Esper provides multiple choices for representing an event. There is no absolute need for you to create new types to represent an event.

Event representations have the following in common:

  • All event representations support nested, indexed and mapped properties (aka. property expression), as explained in more detail below. There is no limitation to the nesting level.

  • All event representations provide event type metadata. This includes type metadata for nested properties.

  • All event representations allow transposing the event itself and parts of all of its property graph into new events. The term transposing refers to selecting the event itself or event properties that are themselves nestable property graphs, and then querying the event's properties or nested property graphs in further statements. The Apache Axiom event representation is an exception and does not currently allow transposing event properties but does allow transposing the event itself.

  • The object and Map representations allow supertypes.

The API behavior for all event representations is the same, with minor exceptions noted in this chapter.

The benefits of multiple event representations are:

  • For applications that already have events in one of the supported representations, there is no need to transform events into an object before processing.

  • Event representations are exchangeable, reducing or eliminating the need to change statements when the event representation changes.

  • Event representations are interoperable, allowing all event representations to interoperate in same or different statements.

  • The choice makes its possible to consciously trade-off performance, ease-of-use, the ability to evolve and effort needed to import or externalize events and use existing event type metadata.

2.2. Event Properties

Event properties capture the state information for an event. Event properties be simple as well as indexed, mapped and nested event properties. The table below outlines the different types of properties and their syntax in an event expression. This syntax allows statements to query deep objects graphs, XML structures and Map events.

Table 2.2. Types of Event Properties

TypeDescriptionSyntaxExample
SimpleA property that has a single value that may be retrieved.
name
sensorId
IndexedAn indexed property stores an ordered collection of objects (all of the same type) that can be individually accessed by an integer-valued, non-negative index (or subscript).
name[index]
sensor[0]
MappedA mapped property stores a keyed collection of objects (all of the same type).
name('key')
sensor('light')
NestedA nested property is a property that lives within another property of an event.
name.nestedname
sensor.value

Combinations are also possible. For example, a valid combination could be person.address('home').street[0].

2.2.1. Escape Characters

If your application uses System.Collection.Generic.IDictionary or XML to represent events, then event property names may themselves contain the dot ('.') character. The backslash ('\') character can be used to escape dot characters in property names, allowing a property name to contain dot characters.

For example, the EPL as shown below expects a property by name part1.part2 to exist on event type MyEvent:

select part1\.part2 from MyEvent

Sometimes your event properties may overlap with EPL language keywords. In this case you may use the backwards apostrophe ` character to escape the property name.

The next example assumes a Quote event that has a property by name order, while order is also a reserved keyword:

select `order` from Quote

2.3. Dynamic Event Properties

Dynamic (unchecked) properties are event properties that need not be known at statement compilation time. Such properties are resolved during runtime: they provide duck typing functionality.

The idea behind dynamic properties is that for a given underlying event representation we don't always know all properties in advance. An underlying event may have additional properties that are not known at statement compilation time, that we want to query on. The concept is especially useful for events that represent rich, object-oriented domain models.

The syntax of dynamic properties consists of the property name and a question mark. Indexed, mapped and nested properties can also be dynamic properties:

Table 2.3. Types of Event Properties

TypeSyntax
Dynamic Simple
name?
Dynamic Indexed
name[index]?
Dynamic Mapped
name('key')?
Dynamic Nested
name?.nestedPropertyName

Dynamic properties always return the System.Object type. Also, dynamic properties return a null value if the dynamic property does not exist on events processed at runtime.

As an example, consider an OrderEvent event that provides an "item" property. The "item" property is of type Object and holds a reference to an instance of either a Service or Product.

Assume that both Service and Product classes provide a property named "price". Via a dynamic property we can specify a query that obtains the price property from either object (Service or Product):

select item.price? from OrderEvent

As a second example, assume that the Service class contains a "serviceName" property that the Product class does not possess. The following query returns the value of the "serviceName" property for Service objects. It returns a null-value for Product objects that do not have the "serviceName" property:

select item.serviceName? from OrderEvent

Consider the case where OrderEvent has multiple implementation classes, some of which have a "timestamp" property. The next query returns the timestamp property of those implementations of the OrderEvent interface that feature the property:

select timestamp? from OrderEvent

The query as above returns a single column named "timestamp?" of type Object.

When dynamic properties are nested, then all properties under the dynamic property are also considered dynamic properties. In the below example the query asks for the "direction" property of the object returned by the "detail" dynamic property:

select detail?.direction from OrderEvent

Above is equivalent to:

select detail?.direction? from OrderEvent

The functions that are often useful in conjunction with dynamic properties are:

  • The cast function casts the value of a dynamic property (or the value of an expression) to a given type.

  • The exists function checks whether a dynamic property exists. It returns true if the event has a property of that name, or false if the property does not exist on that event.

  • The instanceof function checks whether the value of a dynamic property (or the value of an expression) is of any of the given types.

  • The typeof function returns the string type name of a dynamic property.

Dynamic event properties work with all event representations outlined next: Objects, Map-based and XML DOM-based events.

2.4. Fragment and Fragment Type

Sometimes an event can have properties that are itself events. Esper uses the term fragment and fragment type for such event pieces. The best example is a pattern that matches two or more events and the output event contains the matching events as fragments. In other words, output events can be a composite event that consists of further events, the fragments.

Fragments have the same metadata available as their enclosing composite events. The metadata for enclosing composite events contains information about which properties are fragments, or have a property value that can be represented as a fragment and therefore as an event itself.

Fragments and type metadata can allow your application to navigate composite events without the need for using the reflection API and reducing the coupling to the underlying event representation. The API is further described in Section 12.5, “Event and Event Type”.

2.5. Object Events

Object events are object instances that expose properties. While properties are the preferred method for exposing information through Esper, properties can also be exposed if an accessor-style or accessor-method is defined via configuration.

Esper supports event types that extend a superclass or implement one or more interfaces. Also, Esper event pattern and EPL statements can refer to interface classes and abstract classes.

Classes that represent events should be made immutable. As events are recordings of a state change or action that occurred in the past, the relevant event properties should not be changeable. However this is not a hard requirement and the Esper engine accepts events that are mutable as well.

The GetHashCode and Equals methods do not need to be implemented. The implementation of these methods by an event type does not affect the behavior of the engine in any way.

Please see Chapter 13, Configuration on options for naming event types represented by object event types. Types that do not follow standard property conventions, such as those that expose public fields, require additional configuration. Via configuration it is also possible to control case sensitivity in property name resolution. The relevant section in the chapter on configuration is Section 13.4.1.3, “Legacy Event Classes”.

2.5.1. Object Event Properties

As outlined earlier, the different property types are supported by the standard CLR properties specification, and some of which are uniquely supported by Esper:

  • Simple properties have a single value that may be retrieved. The underlying property type might be a CLR primitive (such as int, a simple object (such as a System.String), or a more complex object whose class is defined either by the CLR type structure, by the application, or by a type library included with the application.

  • Indexed - An indexed property stores an ordered collection of objects (all of the same type) that can be individually accessed by an integer-valued, non-negative index (or subscript).

  • Mapped - Esper considers any property that accepts a String-valued key a mapped property.

  • Nested - A nested property is a property that lives within another object which itself is a property of an event.

Assume there is an NewEmployeeEvent event type as shown below. The mapped and indexed properties in this example return objects but could also return CLR primitive types (such as Int32 or String). The Address object and Employee can themselves have properties that are nested within them, such as a street name in the Address object or a name of the employee in the Employee object.

public class NewEmployeeEvent {
	public String FirstName { get; }
	public Address GetAddress(String type);
	public Employee GetSubordinate(int index);
	public Employee[] AllSubordinates { get; }
}

Simple event properties require a read accessible type property returns the property value. This is standard convention for CLR-based applications and your specific language bindings will vary. In this example, the FirstName property returns the firstName event property of type String.

Indexed event properties require either one of the following getter-methods. A method that takes an integer-type key value and returns the property value, such as the GetSubordinate method, or a method that returns an array-type, or a class that implements IEnumerable. An example is the GetSubordinates getter method, which returns an array of Employee but could also return an Iterable. In an EPL or event pattern statement, indexed properties are accessed via the property[index] syntax.

Mapped event properties require a getter-method that takes a String-typed key value and returns the property value, such as the GetAddress method. In an EPL or event pattern statement, mapped properties are accessed via the property('key') syntax.

Nested event properties require a getter-method that returns the nesting object. The Address property and GetSubordinate method are mapped and indexed properties that return a nesting object. In an EPL or event pattern statement, nested properties are accessed via the property.NestedProperty syntax.

All event pattern and EPL statements allow the use of indexed, mapped and nested properties (or a combination of these) anywhere where one or more event property names are expected. The below example shows different combinations of indexed, mapped and nested properties in filters of event pattern expressions (each line is a separate EPL statement):

every NewEmployeeEvent(FirstName='myName')
every NewEmployeeEvent(Address('home').StreetName='Park Avenue')
every NewEmployeeEvent(Subordinate[0].Name='anotherName')
every NewEmployeeEvent(AllSubordinates[1].Name='thatName')
every NewEmployeeEvent(Subordinate[0].Address('home').StreetName='Water Street')

Similarly, the syntax can be used in EPL statements in all places where an event property name is expected, such as in select lists, where-clauses or join criteria.

select FirstName, Address('work'), Subordinate[0].Name, Subordinate[1].Name
from NewEmployeeEvent
where Address('work').StreetName = 'Park Ave'

2.5.2. Property Names

Property names are the result of a mixture of CLR standards and light method introspection of the type. Any properties that are defined via the CLR are automatically exposed. Additionally, any methods that are defined using the naming convention GetPropertyName is exposed as a property. In addition, Esper configuration provides a flag to turn off case-sensitive property names. A sample list of getter methods and property names is:

Table 2.4. Getter Methods and Property Names

MethodProperty NameExample
GetPrice()price
select Price from MyEvent
GetNAME()NAME
select NAME from MyEvent
GetItemDesc()itemDesc
select ItemDesc from MyEvent
GetQ()q
select Q from MyEvent
GetQN()QN
select QN from MyEvent
GetS()s
select S from MyEvent

It is important to note that Java and CLR property conventions are not compatible. In most cases, case conventions are opposite (e.g. Java uses camel case, whereas the CLR uses Pascal case).

2.5.3. Constants and Enumeration

Public static or const fields may also participate in expressions of all kinds, as this example shows:

select * from MyEvent where property=MyConstantClass.FIELD_VALUE

Event properties that are enumeration values can be compared by their enumeration value:

select * from MyEvent where enumProp=EnumClass.ENUM_VALUE_1

Alternatively, a static method may be employed on a class, such as the enumeration class 'EnumClass' as below:

select * from MyEvent where enumProp=EnumClass.valueOf('ENUM_VALUE_1')

If your application does not import, through configuration, the package that contains the enumeration type, then it must also specify the namespace of the type. Enumeration types that are inner classes must be qualified with + following conventions.

For example, the Color enumeration as an inner class to MyEvent in package org.myorg can be referenced as shown:

select * from MyEvent(enumProp=org.myorg.MyEvent+Color.GREEN).std:firstevent()

Instance methods may also be invoked on event instances by specifying a stream name, as shown below:

select myevent.computeSomething() as result from MyEvent as myevent

Chaining instance methods is supported as this example shows:

select myevent.GetComputerFor('books', 'movies').Calculate() as result 
from MyEvent as myevent

2.5.4. Parameterized Types

When your getter methods or accessor fields return a parameterized type, for example IEnumerable<MyEventData> for an indexed property or IDictionary<String, MyEventData> for a mapped property, then property expressions may refer to the properties available through the class that is the type parameter.

An example event that has properties that are parameterized types is:

public class NewEmployeeEvent {
  public String Name { get; }
  public IEnumerable<EducationHistory> Education { get; }
  public IDictionary<String, Address> Addresses { get; }
}

A sample of valid property expressions for this event is shown next:

select Name, Education, Education[0].Date, Addresses('home').Street
from NewEmployeeEvent

2.5.5. Known Limitations

Esper employs byte code generation for fast access to event properties. When byte code generation is unsuccessful, the engine logs a warning and uses CLR reflection to obtain property values instead.

A known limitation is that when an interface has an attribute of a particular type and the actual type returns a subclass of that attribute, the engine logs a warning and uses reflection for that property.

2.6. Map Events

2.6.1. Overview

Events can also be represented by objects that implement the System.Collection.IDictionary<String,Object> interface. Henceforth, we will refer to this type loosely as a DataMap. Event properties of DataMap events are the values in the map accessible through the indexer exposed by the IDictionary interface.

The Map event type is a comprehensive type system that can eliminate the need to use CLR types as event types, thereby making it easier to change types at runtime or generate type information from another source.

A given Map event type can have one or more supertypes that must also be Map event types. All properties available on any of the Map supertypes are available on the type itself. In addition, anywhere within EPL that an event type name of a Map supertype is used, any of its Map subtypes and their subtypes match that expression.

Your application can add properties to an existing Map event type during runtime using the configuration operation UpdateMapEventType. Properties may not be updated or deleted - properties can only be added, and nested properties can be added as well. The runtime configuration also allows removing Map event types and adding them back with new type information.

After your application configures a Map event type by providing a type name, the type name can be used when defining further Map event types by specifying the type name as a property type or an array property type.

One-to-Many relationships in Map event types are represented via arrays. A property in a Map event type may be an array of primitive, an array of object or an array of Map.

The engine can process DataMap events via the SendEvent(DataMap map, String eventTypeName) method on the EPRuntime interface. Entries in the Map represent event properties. Keys must be of type System.String for the engine to be able to look up event property names specified by pattern or EPL statements.

The engine does not validate Map event property names or values. Your application should ensure that objects passed in as event properties match the create schema property names and types, or the configured event type information when using runtime or static configuration.

2.6.2. Map Properties

Map event properties can be of any type. Map event properties that are application objects or that are of type DataMap (or arrays thereof) offer additional power:

  • Properties that are application objects can be queried via the nested, indexed, mapped and dynamic property syntax as outlined earlier.

  • Properties that are of type Map allow Maps to be nested arbitrarily deep and thus can be used to represent complex domain information. The nested, indexed, mapped and dynamic property syntax can be used to query Maps within Maps and arrays of Maps within Maps.

In order to use Map events, the event type name and property names and types must be made known to the engine via Configuration. Please see the examples in Section 13.4.2, “Events represented by System.Collection.Generic.IDictionary”.

The code snippet below creates and processes a Map event. It defines a CarLocationUpdateEvent event type first:

var mapEvent = new Dictionary<string,object>();
mapEvent["carId"] = carId;
mapEvent["direction"] = direction;
epRuntime.SendEvent(mapEvent, "CarLocUpdateEvent");

The CarLocUpdateEvent can now be used in a statement:

select carId from CarLocUpdateEvent.win:time(1 min) where direction = 1

The engine can also query objects as values in a Map event via the nested property syntax. Thus Map events can be used to aggregate multiple data structures into a single event and query the composite information in a convenient way. The example below demonstrates a Map event with a transaction and an account object.

var mapEvent = new Dictionary<string,object>();
mapEvent["txn"] = txn;
mapEvent["account"] = account;
epRuntime.SendEvent(mapEvent, "TxnEvent");

An example statement could look as follows.

select account.id, account.rate * txn.amount 
from TxnEvent.win:time(60 sec) 
group by account.id

2.6.3. Map Supertypes

Your Map event type may declare one or more supertypes when configuring the type at engine initialization time or at runtime through the administrative interface.

Supertypes of a Map event type must also be Map event types. All property names and types of a supertype are also available on a subtype and override such same-name properties of the subtype. In addition, anywhere within EPL that an event type name of a Map supertype is used, any of its Map subtypes also matches that expression (similar to the concept of interface).

This example assumes that the BaseUpdate event type has been declared and acts as a supertype to the AccountUpdate event type (both Map event types):

epService.EPAdministrator.GetConfiguration().
    AddEventType("AccountUpdate", accountUpdateDef, 
    new String[] {"BaseUpdate"});

Your application EPL statements may select BaseUpdate events and receive both BaseUpdate and AccountUpdate events, as well as any other subtypes of BaseUpdate and their subtypes.

// Receive BaseUpdate and any subtypes including subtypes of subtypes
select * from BaseUpdate

Your application Map event type may have multiple supertypes. The multiple inheritance hierarchy between Maps can be arbitrarily deep, however cyclic dependencies are not allowed. If using runtime configuration, supertypes must exist before a subtype to a supertype can be added.

See Section 13.4.2, “Events represented by System.Collection.Generic.IDictionary” for more information on configuring Map event types.

2.6.4. Advanced Map Property Types

2.6.4.1. Nested Properties

Strongly-typed nested Map-within-Map events can be used to build rich, type-safe event types on the fly. Use the AddEventType method on Configuration or ConfigurationOperations for initialization-time and runtime-time type definition.

Noteworthy points are:

  • Objects can appear as properties in Map-within-Map.

  • One may represent Map-within-Map and Map-Array within Map using the name of a previously registered Map event type.

  • There is no limit to the number of nesting levels.

  • Dynamic properties can be used to query Map-within-Map keys that may not be known in advance.

  • The engine returns a null value for properties for which the access path into the nested structure cannot be followed where map entries do not exist.

For demonstration, in this example our top-level event type is an AccountUpdate event, which has an UpdatedField structure as a property. Inside the UpdatedField structure the example defines various fields, as well as a property by name 'history' that holds a type UpdateHistory to represent the update history for the account. The code snippet to define the event type is thus:

IDictionary<String, Object> updatedFieldDef = new Dictionary<String, Object>();
updatedFieldDef["name"] = typeof(string);
updatedFieldDef["addressLine1"] = typeof(string);
updatedFieldDef["history"] = typeof(UpdateHistory);

IDictionary<String, Object> accountUpdateDef = new Dictionary<String, Object>();
accountUpdateDef["accountId"] = typeof(long);
accountUpdateDef["fields"] = updatedFieldDef;

epService.EPAdministrator.GetConfiguration().
    AddEventType("AccountUpdate", accountUpdateDef);

The next code snippet populates a sample event and sends the event into the engine:

IDictionary<String, Object> updatedField = new Dictionary<String, Object>();
updatedField["name"] = "Joe Doe";
updatedField["addressLine1"] = "40 Popular Street";
updatedField["history"] = new UpdateHistory();

IDictionary<String, Object> accountUpdate = new Dictionary<String, Object>();
accountUpdate["accountId"] = 10009901;
accountUpdate["fields"] = updatedField;

epService.EPRuntime.SendEvent(accountUpdate, "AccountUpdate");

Last, a sample query to interrogate AccountUpdate events is as follows:

select accountId, fields.name, fields.addressLine1, fields.history.lastUpdate
from AccountUpdate

Note that type information for nested maps is only available to the immediately selecting stream. For example, the second select-query does not work:

insert into MyStream select fields from NestedMapEvent
// this does not work ... instead select the individual fields in the insert-into statement
select fields.name from MyStream 

2.6.4.2. Map Event Type Properties

Your application may declare a Map event type for reuse within other Map event types or for one-to-many properties represented by an array of Maps.

This example declares a Map event type by name AmountCurrency with amount and currency properties:

IDictionary<String, Object> amountAndCurr = new Dictionary<String, Object>();
amountAndCurr["amount"] = typeof(double);
amountAndCurr["currency"] = typeof(string);

epService.EPAdministrator.GetConfiguration().
    AddEventType("AmountCurrency", amountAndCurr);

The AmountCurrency type is now available for use as a property type itself. Below code snippet declares OrderItem to hold an item number and AmountCurrency:

IDictionary<String, Object> orderItem = new Dictionary<String, Object>();
orderItem["itemNum"] = typeof(int);
orderItem["price"] = "AmountCurrency";    // The property type is the name itself

epService.EPAdministrator.GetConfiguration().
    AddEventType("OrderItem", orderItem);

2.6.4.3. One-to-Many Relationships

To model repeated properties within a Map, you may use arrays as properties in a Map. You may use an array of primitive types or an array of objects or an array of a previously declared Map event type.

When using a previously declared Map event type as an array property, the literal [] must be appended after the event type name.

This following example defines a Map event type by name Sale to hold array properties of the various types. It assumes a SalesPerson type exists and a Map event type by name OrderItem was declared:

Map<String, Object> sale = new HashMap<String, Object>();
sale["userids"] = typeof(int[]);
sale["salesPersons"] = typeof(SalesPerson[]);
sale["items"] = "OrderItem[]";	 // The property type is the name itself appended by []

epService.EPAdministrator.GetConfiguration().
    AddEventType("SaleEvent", sale);

The three properties that the above example declares are:

  • An integer array of user ids.

  • An array of SalesPerson objects.

  • An array of Maps for order items.

The next EPL statement is a sample query asking for property values held by arrays:

select userids[0], salesPersons[1].name, 
    items[1], items[1].price.amount from SaleEvent

2.7. System.Xml.XmlNode XML Events

Events can be represented as System.Xml.XmlNode instances and send into the engine via the SendEvent method on EPRuntime or via EventSender. Please note that configuration is required so the event type name and root element name is known. See Chapter 13, Configuration.

If a XML schema document (XSD file) can be made available as part of the configuration, then Esper can read the schema and appropriately present event type metadata and validate statements that use the event type and its properties. See Section 2.7.1, “Schema-Provided XML Events”.

When no XML schema document is provided, XML events can still be queried, however the return type and return values of property expressions are string-only and no event type metadata is available other then for explicitly configured properties. See Section 2.7.2, “No-Schema-Provided XML Events”.

In all cases Esper allows you to configure explicit XPath expressions as event properties. You can specify arbitrary XPath functions or expressions and provide a property name and type by which result values will be available for use in EPL statements. See Section 2.7.3, “Explicitly-Configured Properties”.

Nested, mapped and indexed event properties are also supported in expressions against System.Xml.XmlNode events. Thus XML trees can conveniently be interrogated via the property expression syntax.

Only one event type per root element name may be configured. The engine recognizes each event by its root element name or you may use EventSender to send events.

This section uses the following XML document as an example:

<?xml version="1.0" encoding="UTF-8"?>
<Sensor xmlns="SensorSchema">
  <ID>urn:epc:1:4.16.36</ID>
  <Observation Command="READ_PALLET_TAGS_ONLY">
    <ID>00000001</ID>
    <Tag>
      <ID>urn:epc:1:2.24.400</ID>
    </Tag>
    <Tag>
      <ID>urn:epc:1:2.24.401</ID>
    </Tag>
  </Observation>
</Sensor>

The schema for the example is:

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

  <xs:element name="Sensor">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="ID" type="xs:string"/>
        <xs:element ref="Observation" />
      </xs:sequence>
    </xs:complexType>
  </xs:element>

  <xs:element name="Observation">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="ID" type="xs:string"/>
        <xs:element ref="Tag" maxOccurs="unbounded" />
      </xs:sequence>
      <xs:attribute name="Command" type="xs:string" use="required" />
    </xs:complexType>
  </xs:element>

  <xs:element name="Tag">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="ID" type="xs:string"/>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
</xs:schema>

2.7.1. Schema-Provided XML Events

If you have a XSD schema document available for your XML events, Esper can interrogate the schema. The benefits are:

  • New EPL statements that refer to event properties are validated against the types provided in the schema.

  • Event type metadata becomes available for retrieval as part of the EventType interface.

2.7.1.1. Getting Started

The engine reads a XSD schema file from an URL you provide. Make sure files imported by the XSD schema file can also be resolved.

The configuration accepts a schema URL. This is a sample code snippet to determine a schema URL from the resource manager:

URL schemaURL = ResourceManager.ResolveResourceURL("sensor.xsd");

Here is a sample use of the runtime configuration API, please see Chapter 13, Configuration for further examples.

epService = EPServiceProviderManager.GetDefaultProvider();
ConfigurationEventTypeXMLDOM sensorcfg = new ConfigurationEventTypeXMLDOM();
sensorcfg.RootElementName = "Sensor");
sensorcfg.SchemaResource = schemaURL.ToString();
epService.EPAdministrator.GetConfiguration()
    .AddEventType("SensorEvent", sensorcfg);

You must provide a root element name. This name is used to look up the event type for the SendEvent(System.Xml.XmlNode node) method. An EventSender is a useful alternative method for sending events if the type lookup based on the root or document element name is not desired.

After adding the event type, you may create statements and send events. Next is a sample statement:

select ID, Observation.Command, Observation.ID, 
  Observation.Tag[0].ID, Observation.Tag[1].ID
from SensorEvent

As you can see from the example above, property expressions can query property values held in the XML document's elements and attributes.

There are multiple ways to obtain a XML DOM document instance from a XML string. The next code snippet shows how to obtain a XML DOM System.Xml.XmlDocument instance:

XmlDocument doc = new XmlDocument();
doc.LoadXml(xml);

Send the System.Xml.XmlNode or Document object into the engine for processing:

epService.EPRuntime.SendEvent(doc);

2.7.1.2. Property Expressions and Namespaces

By default, property expressions such as Observation.Tag[0].ID are evaluated by a fast DOM-walker implementation provided by Esper. This DOM-walker implementation is not namespace-aware.

Should you require namespace-aware traversal of the DOM document, you must set the xpath-property-expr configuration option to true (default is false). This flag causes Esper to generate namespace-aware XPath expressions from each property expression instead of the DOM-walker, as described next. Setting the xpath-property-expr option to true requires that you also configure namespace prefixes as described below.

When matching up the property names with the XSD schema information, the engine determines whether the attribute or element provides values. The algorithm checks attribute names first followed by element names. It takes the first match to the specified property name.

2.7.1.3. Property Expression to XPath Rewrite

By setting the xpath-property-expr option the engine rewrites each property expression as an XPath expression, effectively handing the evaluation over to the underlying XPath implementation available from classpath. Most JVM have a built-in XPath implementation and there are also optimized, fast implementations such as Jaxen that can be used as well.

Set the xpath-property-expr option if you need namespace-aware document traversal, such as when your schema mixes several namespaces and element names are overlapping.

The below table samples several property expressions and the XPath expression generated for each, without namespace prefixes to keep the example simple:

Table 2.5. Property Expression to XPath Expression

Property ExpressionEquivalent XPath
Observeration.ID/Sensor/Observation/ID
Observeration.Command/Sensor/Observation/@Command
Observeration.Tag[0].ID/Sensor/Observation/Tag[position() = 1]/ID

For mapped properties that are specified via the syntax name('key'), the algorithm looks for an attribute by name id and generates a XPath expression as mapped[@id='key'].

Finally, here is an example that includes all different types of properties and their XPath expression equivalent in one property expression:

select nested.mapped('key').indexed[1].attribute from MyEvent

The equivalent XPath expression follows, this time including n0 as a sample namespace prefix:

/n0:rootelement/n0:nested/n0:mapped[@id='key']/n0:indexed[position() = 2]/@attribute

2.7.1.4. Array Properties

All elements that are unbound or have max occurs greater then 1 in the XSD schema are represented as indexed properties and require an index for resolution.

For example, the following is not a valid property expression in the sample Sensor document: Observeration.Tag.ID. As no index is provided for Tag, the property expression is not valid.

Repeated elements within a parent element in which the repeated element is a simple type also are represented as an array.

Consider the next XML document:

<item>
  <book sku="8800090">
    <author>Isaac Asimov</author>
    <author>Robert A Heinlein</author>
  </book>
</item>

Here, the result of the expression book.author is an array of type String and the result of book.author[0] is a String value.

2.7.1.5. Dynamic Properties

Dynamic properties are not validated against the XSD schema information and their result value is always System.Xml.XmlNode. You may use a user-defined function to process dynamic properties returning Node. As an alternative consider using an explicit property.

An example dynamic property is Origin?.ID which will look for an element by name Origin that contains an element or attribute node by name LocationCode:

select Origin?.LocationCode from SensorEvent

2.7.1.6. Transposing Properties

When providing a XSD document, the default configuration allows to transpose property values that are themselves complex elements, as defined in the XSD schema, into a new stream. This behavior can be controlled via the flag auto-fragment.

For example, consider the next query:

insert into ObservationStream
select ID, Observation from SensorEvent

The Observation as a property of the SensorEvent gets itself inserted into a new stream by name ObservationStream. The ObservationStream thus consists of a string-typed ID property and a complex-typed property named Observation, as described in the schema.

A further statement can use this stream to query:

select Observation.Command, Observation.Tag[0].ID from ObservationStream

Before continuing the discussion, here is an alternative syntax using the wildcard-select, that is also useful:

insert into TagListStream
select ID as sensorId, Observation.* from SensorEvent

The new TagListStream has a string-typed ID and Command property as well as an array of Tag properties that are complex types themselves as defined in the schema.

Next is a sample statement to query the new stream:

select sensorId, Command, Tag[0].ID from TagListStream

Please note the following limitations:

  • The XPath standard prescribes that XPath expressions against System.Xml.Node are evaluated against the owner document of the Node. Therefore XPath is not relative to the current node but absolute against each node's owner document. Since Esper does not create new document instances for transposed nodes, transposing properties is not possible when the xpath-property-expr flag is set.

  • Complex elements that have both simple element values and complex child elements are not transposed. This is to ensure their property value is not hidden. Use an explicit XPath expression to transpose such properties.

Esper automatically registers a new event type for transposed properties. It generates the type name of the new XML event type from the XML event type name and the property names used in the expression. The synposis is type_name.property_name[.property_name...]. The type name can be looked up, for example for use with EventSender or can be created in advance.

2.7.1.7. Event Sender

An IEventSender sends events into the engine for a given type, saving a type lookup based on element name.

This brief example sends an event via IEventSender:

EventSender sender = epRuntime.GetEventSender("SensorEvent");
sender.sendEvent(node);

The XML DOM event sender checks the root element name before processing the event. Use the event-sender-validates-root setting to disable validation. This forces the engine to process XML documents according to any predefined type without validation of the root element name.

2.7.2. No-Schema-Provided XML Events

Without a schema document a XML event may still be queried. However there are important differences in the metadata available without a schema document and therefore the property expression results. These differences are outlined below.

All property expressions against a XML type without schema are assumed valid. There is no validation of the property expression other then syntax validation. At runtime, property expressions return string-type values or null if the expression did not yield a matching element or attribute result.

When asked for property names or property metadata, a no-schema type returns empty array.

In all other aspects the type behaves the same as the schema-provided type described earlier.

2.7.3. Explicitly-Configured Properties

Regardless of whether or not you provide a XSD schema for the XML event type, you can always fall back to configuring explicit properties that are backed by XPath expressions.

For further documentation on XPath, please consult the XPath standard or other online material.

2.7.3.1. Simple Explicit Property

Shown below is an example configuration that adds an explicit property backed by a XPath expression and that defines namespace prefixes:

epService = EPServiceProviderManager.GetDefaultProvider();
ConfigurationEventTypeXMLDOM sensorcfg = new ConfigurationEventTypeXMLDOM();
sensorcfg.AddXPathProperty("countTags", "count(/ss:Sensor/ss:Observation/ss:Tag)", 
    XPathResultType.Number);
sensorcfg.AddNamespacePrefix("ss", "SensorSchema");
sensorcfg.RootElementName = "Sensor";
epService.EPAdministrator.GetConfiguration()
    .AddEventType("SensorEvent", sensorcfg);

The countTags property is now available for querying:

select countTags from SensorEvent

The XPath expression count(...) is a XPath built-in function that counts the number of nodes, for the example document the result is 2.

2.7.3.2. Explicit Property Casting and Parsing

Esper can parse or cast the result of your XPath expression to the desired type. Your property configuration provides the type to cast to, like this:

sensorcfg.AddXPathProperty("countTags", "count(/ss:Sensor/ss:Observation/ss:Tag)", 
    XPathResultType.Number, "int");

The type supplied to the property configuration must be one of the built-in types. Arrays of built-in type are also possible, requiring the XPathResultType.NodeSet type returned by your XPath expression, as follows:

sensorcfg.AddXPathProperty("idarray", "//ss:Tag/ss:ID", 
    XPathResultType.NodeSet, "String[]");

The XPath expression //ss:Tag/ss:ID returns all ID nodes under a Tag node, regardless of where in the node tree the element is located. For the example document the result is 2 array elements urn:epc:1:2.24.400 and urn:epc:1:2.24.40.

2.7.3.3. Node and Nodeset Explicit Property

An explicit property may return XPathResultType.Any or XPathResultType.NodeSet and can provide the event type name of a pre-configured event type for the property. The method name to add such properties is AddXPathPropertyFragment.

This code snippet adds two explicit properties and assigns an event type name for each property:

sensorcfg.AddXPathPropertyFragment("tagOne", "//ss:Tag[position() = 1]", 
    XPathResultType.Any, "TagEvent");
sensorcfg.AddXPathPropertyFragment("tagArray", "//ss:Tag", 
    XPathResultType.NodeSet, "TagEvent");

The configuration above references the TagEvent event type. This type must also be configured. Prefix the root element name with "//" to cause the lookup to search the nested schema elements for the definition of the type:

ConfigurationEventTypeXMLDOM tagcfg = new ConfigurationEventTypeXMLDOM();
tagcfg.RootElementName = "//Tag";
tagcfg.SchemaResource = schemaURL;
epAdministrator.GetConfiguration()
    .AddEventType("TagEvent", tagcfg);

The tagOne and tagArray properties are now ready for selection and transposing to further streams:

insert into TagOneStream select tagOne.* from SensorEvent

Select from the new stream:

select ID from TagOneStream

An example with indexed properties is shown next:

insert into TagArrayStream select tagArray as mytags from SensorEvent

Select from the new stream:

select mytags[0].ID from TagArrayStream

2.8. Additional Event Representations

Part of the extension and plug-in features of Esper is an event representation API. This set of classes allow an application to create new event types and event instances based on information available elsewhere, statically or dynamically at runtime when EPL statements are created. Please see Section 15.8, “Event Type And Event Object” for details.

Creating a plug-in event representation can be useful when your application has existing types that carry event metadata and event property values and your application does not want to (or cannot) extract or transform such event metadata and event data into one of the built-in event representations (CLR objects, Map or XML DOM).

Further use of a plug-in event representation is to provide a faster or short-cut access path to event data. For example, access to event data stored in a XML format through the Streaming API for XML (StAX) is known to be very efficient. A plug-in event representation can also provide network lookup and dynamic resolution of event type and dynamic sourcing of event instances.

The chapter on Section 15.8, “Event Type And Event Object” explains how to create your own custom event representation.

2.9. Updating, Merging and Versioning Events

To summarize, an event is an immutable record of a past occurrence of an action or state change, and event properties contain useful information about an event.

The length of time an event is of interest to the event processing engine (retention time) depends on your EPL statements, and especially the data window, pattern and output rate limiting clauses of your statements.

During the retention time of an event more information about the event may become available, such as additional properties or changes to existing properties. Esper provides three concepts for handling updates to events.

The first means to handle updating events is the update istream clause as further described in Section 4.20, “Updating an Insert Stream: the Update IStream Clause”. It is useful when you need to update events as they enter a stream, before events are evaluated by any particular consuming statement to that stream.

The second means to update events is the on-merge and on-update clauses, for use with named windows only, as further described in Section 4.15.9, “Triggered Upsert using the On-Merge Clause” and Section 4.15.7, “Updating Named Windows: the On Update clause”. On-merge is similar to the SQL merge clause and provides what is known as an "Upsert" operation: Update existing events or if no existing event(s) are found then insert a new event, all in one atomic operation provided by a single EPL statement. On-update can be used to update individual properties of events held in a named window.

The third means to handle updating events is the revision event types, for use with named windows only, as further described in Section 4.15.11, “Versioning and Revision Event Type Use with Named Windows”. With revision event types one can declare, via configuration only, multiple different event types and then have the engine present a merged event type that contains a superset of properties of all merged types, and have the engine merge events as they arrive without additional EPL statements.

Note that patterns do not reflect changes to past events. For the temporal nature of patterns, any changes to events that were observed in the past do not reflect upon current pattern state.

2.10. Coarse-Grained Events

Your application events may consist of fairly comprehensive, coarse-grained structures or documents. For example in business-to-business integration scenarios, XML documents or other event objects can be rich deeply-nested graphs of event properties.

To extract information from a coarse-grained event or to perform bulk operations on the rows of the property graph in an event, Esper provides a convenient syntax: When specifying a filter expression in a pattern or in a select clause, it may contain a contained-event selection syntax, as further described in Section 4.19, “Contained-Event Selection”.

2.11. Event Objects Populated by Insert Into

The insert into clause can populate plain-old object events and IDictionary events directly from the results of select clause expressions. Simply use the event type name as the stream name in the insert into clause as described in Section 4.10, “Merging Streams and Continuous Insertion: the Insert Into Clause”.

The column names specified in the select and insert into clause must match available writable properties in the event object to be populated (the target event type). The expression result types of any expressions in the select clause must also be compatible with the property types of the target event type.

Consider the following example statement:

insert into com.mycompany.NewEmployeeEvent 
select fname as firstName, lname as lastName from HRSystemEvent

The above example specifies the fully-qualified class name of NewEmployeeEvent. The engine instantianes NewEmployeeEvent for each result row and populates the firstName and lastName properties of each instance from the result of select clause expressions. The HRSystemEvent in the example is assumed to have lname and fname properties.

Note how the example uses the as-keyword to assign column names that match the property names of the NewEmployeeEvent target event. If the property names of the source and target events are the same, the as-keyword is not required.

The next example is an alternate form and specifies property names within the insert into clause instead. The example also assumes that NewEmployeeEvent has been defined or imported via configuration since it does not specify the event class package name:

insert into NewEmployeeEvent(firstName, lastName) 
select fname, lname from HRSystemEvent

Finally, this example populates HRSystemEvent events. The example populates the value of a type property where the event has the value 'NEW' and populates a new event object with the value 'HIRED', copying the fname and lname property values to the new event object:

insert into HRSystemEvent 
select fname, lname, 'HIRED' as type from HRSystemEvent(type='NEW')

The matching of the select or insert into-clause column names to target event type's property names is case-sensitive. It is allowed to only populate a subset of all available columns in the target event type. Wildcard (*) is also allowed and copies all fields of the events or multiple events in a join.

For CLR object events, your event class must provide writable properties. The event type should also provide a default constructor taking no parameters. If your event class does not have a default constructor, your application may configure a factory method via ConfigurationEventTypeLegacy.

The engine follows CLR standards in terms of widening, performing widening automatically in cases where widening type conversion is allowed without loss of precision, for both boxed and primitive types.

When inserting array-typed properties into a CLR or Map-type underlying event the event definition should declare the target property as an array.

Please note the following limitations:

  • Event types that utilize XML System.Xml.XmlNode underlying event objects cannot be target of an insert into clause.


© 2011 EsperTech Inc. All Rights Reserved