esper.codehaus.org and espertech.comDocumentation

Chapter 19. Examples, Tutorials, Case Studies

19.1. Examples Overview
19.2. Running the Examples
19.3. AutoID RFID Reader
19.4. Runtime Configuration
19.5. JMS Server Shell and Client
19.5.1. Overview
19.5.2. JMS Messages as Events
19.5.3. JMX for Remote Dynamic Statement Management
19.6. Market Data Feed Monitor
19.6.1. Input Events
19.6.2. Computing Rates Per Feed
19.6.3. Detecting a Fall-off
19.6.4. Event generator
19.7. OHLC Plug-in View
19.8. Transaction 3-Event Challenge
19.8.1. The Events
19.8.2. Combined event
19.8.3. Real time summary data
19.8.4. Find problems
19.8.5. Event generator
19.9. Self-Service Terminal
19.9.1. Events
19.9.2. Detecting Customer Check-in Issues
19.9.3. Absence of Status Events
19.9.4. Activity Summary Data
19.9.5. Sample Application for J2EE Application Server
19.10. Assets Moving Across Zones - An RFID Example
19.11. StockTicker
19.12. MatchMaker
19.13. Named Window Query
19.14. Sample Virtual Data Window
19.15. Sample Cycle Detection
19.16. Quality of Service
19.17. Trivia Geeks Club

This chapter outlines the examples that come with Esper in the examples folder of the distribution. Each sample is in a separate folder that contains all files needed by the example, excluding jar files.

Here is an overview over the examples in alphabetical order:

Table 19.1. Examples

NameDescription
Section 19.3, “AutoID RFID Reader”

An array of RFID readers sense RFID tags as pallets are coming within the range of one of the readers.

Shows the use of an XSD schema and XML event representation. A single statement shows a rolling time window, a where-clause filter on a nested property and a group-by.

Section 19.6, “Market Data Feed Monitor”

Processes a raw market data feed and reports throughput statistics and detects when the data rate of a feed falls off unexpectedly.

Demonstrates a batch time window and a rolling time window with a having-clause. Multi-threaded example with a configurable number of threads and a simulator for generating feed data.

Section 19.12, “MatchMaker”

In the MatchMaker example every mobile user has an X and Y location and the task of the event patterns created by this example is to detect mobile users that are within proximity given a certain range, and for which certain properties match preferences.

Dynamically creates and removes event patterns that use range matching based on mobile user events received.

Section 19.13, “Named Window Query”

A mini-benchmark that handles temperature sensor events. The sample creates a named window and fills it with a large number of events. It then executes a large number of pre-defined queries as well as fire-and-forget queries and reports times.

Study this example if you are interested in named windows, Map event type representation, fire-and-forget queries as well as pre-defined queries via on-select, and the performance aspects.

Section 19.14, “Sample Virtual Data Window”

This example demonstrates the use of virtual data window to expose a (large) external data store, without any need to keep events in memory, and without sacrificing query performance.

Section 19.15, “Sample Cycle Detection”

This example showcases the aggregation multi-function extension API for use with a cycle-detection problem detecting cycles in transactions between accounts.

Section 19.7, “OHLC Plug-in View”

A plug-in custom view addressing a problem in the financial space: Computes open-high-low-close bars for minute-intervals of events that may arrive late, based on each event's timestamp.

A custom plug-in view based on the extension API can be a convenient and reusable way to express a domain-specific analysis problem as a unit, and this example includes the code for the OHLC view factory and view as well as simulator to test the view.

Section 20.3, “Using the performance kit”

A benchmark that is further described in the performance section of this document under performance kit.

Section 19.16, “Quality of Service”

This example develops some code for measuring quality-of-service levels such as for a service-level agreement (SLA).

This example combines patterns with select-statements, shows the use of the timer 'at' operator and followed-by operator ->, and uses the iterator API to poll for current results.

Section 19.10, “Assets Moving Across Zones - An RFID Example”

An example out of the RFID domain processes location report events. The example includes a simple Swing-based GUI for visualization allows moving tags from zone to zone visually. It also a contains comprehensive simulator to generate data for a large number of asset groups and their tracking.

The example hooks up statements that aggregate and detect patterns in the aggregated data to determine when an asset group constraint is violated.

Section 19.4, “Runtime Configuration”

Example code to demonstrate various key runtime configuration options such as adding event types on-the-fly, adding new variables, adding plug-in single-row and aggregation functions and adding variant streams and revision type definition.

Section 19.5, “JMS Server Shell and Client”

The server shell is a Java Messaging Service (JMS) -based server and client that send and listens to messages on a JMS destination. It also demonstrates a simple Java Management Extension (JMX) MBean for remote statement management.

A single EPL statement computes an average duration for each IP address on a rolling time window and outputs a snapshot every 2 seconds.

Section 19.11, “StockTicker”

An example from the financial domain that features event patterns to filter stock tick events based on price and symbol. The example is designed to provide a high volume of events and includes multithreaded unit test code as well as a simulting data generator.

Perhaps this is a good example to learn the API and get started with event patterns. The example dynamically creates and removes event patterns based on price limit events received.

Section 19.9, “Self-Service Terminal”

A J2EE-based self-service terminal managing system in an airport that gets a lot of events from connected terminals.

Contains a message-driven bean (EJB-MDB) for use in a J2EE container, a client and a simulator, as well as EPL statements for detecting various conditions. A version that runs outside of a J2EE container is also available.

Section 19.17, “Trivia Geeks Club”

Trivia Geeks Club demonstrates EPL for a scoring system computing scores in a trivia game.


In order to compile and run the samples please follow the below instructions:

Each example also provides Eclipse project .classpath and .project files. The Eclipse projects expect an esper_runtime user library that includes the runtime dependencies.

JUnit tests exist for the example code. The JUnit test source code for the examples can be found in each example's src/test folder. To build and run the example JUnit tests, use the Maven 2 goal test.

In this example an array of RFID readers sense RFID tags as pallets are coming within the range of one of the readers. A reader generates XML documents with observation information such as reader sensor ID, observation time and tags observed. A statement computes the total number of tags per reader sensor ID within the last 60 seconds.

This example demonstrates how XML documents unmarshalled to org.w3c.dom.Node DOM document nodes can natively be processed by the engine without requiring Java object event representations. The example uses an XPath expression for an event property counting the number of tags observed by a sensor. The XML documents follow the AutoID (http://www.autoid.org/) organization standard.

The classes for this example can be found in package com.espertech.esper.example.autoid. As events are XML documents with no Java object representation, the example does not have event classes.

A simulator that can be run from the command line is also available for this example. The simulator generates a number of XML documents as specified by a command line argument and prints out the totals per sensor. Run "run_autoid.bat" (Windows) or "run_autoid.sh" (Unix) to start the AutoID simulator. Please see the readme file in the same folder for build instructions and command line parameters.

The code snippet below shows the simple statement to compute the total number of tags per sensor. The statement is created by class com.espertech.esper.example.autoid.RFIDTagsPerSensorStmt.

select ID as sensorId, sum(countTags) as numTagsPerSensor
from AutoIdRFIDExample.win:time(60 seconds)
where Observation[0].Command = 'READ_PALLET_TAGS_ONLY'
group by ID

This example demonstrates various key runtime configuration options such as adding event types on-the-fly, adding new variables, adding plug-in single-row and aggregation functions and adding variant streams and revision type definition.

The classes for this example live in package com.espertech.esper.example.runtimeconfig.

The server shell is a Java Messaging Service (JMS) -based server that listens to messages on a JMS destination, and sends the received events into Esper. The example also demonstrates a Java Management Extension (JMX) MBean that allows remote dynamic statement management. This server has been designed to run with either Tibco (TM) Enterprise Messaging System (Tibco EMS), or with Apache ActiveMQ, controlled by a properties file.

The server shell has been created as an alternative to the EsperIO Spring JMSTemplate adapter. The server shell is a low-latency processor for byte messages. It employs JMS listeners to process message in multiple threads, this model reduces thread context switching for many JMS providers. The server is configurable and has been tested with two JMS providers. It consists of only 10 classes and is thus easy to understand.

The server shell sample comes with a client (server shell client) that sends events into the JMS-based server, and that also creates a statement on the server remotely through a JMX MBean proxy class.

The server shell classes for this example live in package com.espertech.esper.example.servershell. Configure the server to point to your JMS provider by changing the properties in the file servershell_config.properties in the etc folder. Make sure your JMS provider (ActiveMQ or Tibco EMS) is running, then run "run_servershell.bat" (Windows) or "run_servershell.sh" (Unix) to start the JMS server.

Start the server shell process first before starting the client, since the client also demonstrates remote statement management through JMX by attaching to the server process.

The client classes to the server shell can be found in package com.espertech.esper.example.servershellclient. The client shares the same configuration file as the server shell. Run "run_servershellclient.bat" (Windows) or "run_servershellclient.sh" (Unix) to start the JMS producer client that includes a JMX client as well.

This example processes a raw market data feed. It reports throughput statistics and detects when the data rate of a feed falls off unexpectedly. A rate fall-off may mean that the data is stale and we want to alert when there is a possible problem with the feed.

The classes for this example live in package com.espertech.esper.example.marketdatafeed. Run "run_mktdatafeed.bat" (Windows) or "run_mktdatafeed.sh" (Unix) in the examples/etc folder to start the market data feed simulator.

This example contains a fully-functional custom view based on the extension API that computes OHLC open-high-low-close bars for events that provide a long-typed timestamp and a double-typed value.

OHLC bar is a problem out of the financial domain. The "Open" refers to the first datapoint and the "Close" to the last datapoint in an interval. The "High" refers to the maximum and the "Low" to the minimum value during each interval. The term "bar" is used to describe each interval results of these 4 values.

The example provides an OHLC view that is hardcoded to 1-minute bars. It considers the timestamp value carried by each event, and not the system time. The cutoff time after which an event is no longer considered for a bar is hardcoded to 5 seconds.

The view assumes that events arrive in timestamp order: Each event's timestamp value is equal to or higher then the timestamp value provided by the prior event.

The view may also be used together with std:groupwin to group per criteria, such as symbol. In this case the assumption of timestamp order applies per symbol.

The view gracefully handles no-event and late-event scenarios. Interval boundaries are defined by system time, thus event timestamp and system time must roughly be in-sync, unless using external timer events.

The classes for this example live in package com.espertech.esper.example.transaction. Run "run_txnsim.bat" (Windows) or "run_txnsim.sh" (Unix) to start the transaction simulator. Please see the readme file in the same folder for build instructions and command line parameters.

To make testing easier, standard and to demonstrate how the example works, the example is including an event generator. The generator generates events for a given number of transactions, using the following rules:

To make things harder, we don‘t want transaction events coming in order. This code ensures that they come completely out of order. To do this, we fill in a bucket with events and, when the bucket is full, we shuffle it. The buckets are sized so that some transactions‘ events will be split between buckets. So, you have a fairly randomized flow of events, representing the worst case from a big, distributed infrastructure.

The generator lets you change the size of the bucket (small, medium, large, larger, largerer). The larger the bucket size, the more events potentially come in between two events in a given transaction and so, the more the performance characteristics like buffers, hashes/indexes and other structures are put to the test as the bucket size increases.

The example is about a J2EE-based self-service terminal managing system in an airport that gets a lot of events from connected terminals. The event rate is around 500 events per second. Some events indicate abnormal situations such as 'paper low' or 'terminal out of order'. Other events observe activity as customers use a terminal to check in and print boarding tickets.

The example code in the distribution package implements a message-driven enterprise java bean (MDB EJB). We used an MDB as a convenient place for processing incoming events via a JMS message queue or topic. The example uses 2 JMS queues: One queue to receive events published by terminals, and a second queue to indicate situations detected via EPL statement and listener back to a receiving process.

This example has been packaged for deployment into a JBoss Java application server (see http://www.jboss.org) with default deployment configuration. JBoss is an open-source application server available under LGPL license. Of course the choice of application server does not indicate a requirement or preference for the use of Esper in a J2EE container. Other quality J2EE application servers are available and perhaps more suitable to run this example or a similar application.

The complete example code can be found in the "examples/terminalsvc" folder of the distribution. The standalone version that does not require a J2EE container is in "examples/terminalsvc-jse".

The example also contains an event simulator that generates meaningful events. The simulator can be run from the directory "examples/terminalsvc/etc" via the command "run_terminalsvc_sender.bat" (Windows) and "run_terminalsvc_sender.sh" (Linux). The event simulator generates a batch of at least 200 events every 1 second. Randomly, with a chance of 1 in 10 for each batch of events, the simulator generates either an OutOfOrder or a LowPaper event for a random terminal. Each batch the simulator generates 100 random terminal ids and generates a Checkin event for each. It then generates either a Cancelled or a Completed event for each. With a chance of 1 in 1000, it generates an OutOfOrder event instead of the Cancelled or Completed event for a terminal.

The event receiver listens to the MDB-outcoming queue for alerts and prints these out to console. The receiver can be run from the directory "examples/terminalsvc/etc" via the command "run_terminalsvc_receiver.bat" (Windows) and "run_terminalsvc_receiver.sh" (Linux). Before running please copy the jboss-client.jar file from your JBoss AS installation bin directory to the "terminalsvc/lib" folder.

The receiver and sender code use "guest" as user and "pass" as password. Add the "guest" user using the Jboss "add-user" script and assign the role "guest". Your JBoss server may need to start with "-c standalone-full.xml" to have the messaging subsystem available.

Add queue configurations to the messaging subsystem configuration as follows:

<jms-queue name="queue_a">
  <entry name="queue_a"/>
  <entry name="java:jboss/exported/jms/queue/queue_a"/>
</jms-queue>
<jms-queue name="queue_b">
  <entry name="queue_b"/>
  <entry name="java:jboss/exported/jms/queue/queue_b"/>
</jms-queue>

Disable persistence in the messaging subsystem for this example so we are not running out of disk space:

<persistence-enabled>false</persistence-enabled>

This example out of the RFID domain processes location report events. Each location report event indicates an asset id and the current zone of the asset. The example solves the problem that when a given set of assets is not moving together from zone to zone, then an alert must be fired.

Each asset group is tracked by 2 statements. The two statements to track a single asset group consisting of assets identified by asset ids {1, 2, 3} are as follows:

insert into CountZone_G1
select 1 as groupId, zone, count(*) as cnt
from LocationReport(assetId in 1, 2, 3).std:unique(assetId)
group by zone

select Part.zone from pattern [
  every Part=CountZone_G1(cnt in (1,2)) ->
    (timer:interval(10 sec)  and not CountZone_G1(zone=Part.zone, cnt in (0,3)))]

The classes for this example can be found in package com.espertech.esper.example.rfid.

This example provides a Swing-based GUI that can be run from the command line. The GUI allows drag-and-drop of three RFID tags that form one asset group from zone to zone. Each time you move an asset across the screen the example sends an event into the engine indicating the asset id and current zone. The example detects if within 10 seconds the three assets do not join each other within the same zone, but stay split across zones. Run "run_rfid_swing.bat" (Windows) or "run_rfid_swing.sh" (Unix) to start the example's Swing GUI.

The example also provides a simulator that can be run from the command line. The simulator generates a number of asset groups as specified by a command line argument and starts a number of threads as specified by a command line argument to send location report events into the engine. Run "run_rfid_sim.bat" (Windows) or "run_rfid_sim.sh" (Unix) to start the RFID location report event simulator. Please see the readme file in the same folder for build instructions and command line parameters.

The StockTicker example comes from the stock trading domain. The example creates event patterns to filter stock tick events based on price and symbol. When a stock tick event is encountered that falls outside the lower or upper price limit, the example simply displays that stock tick event. The price range itself is dynamically created and changed. This is accomplished by an event patterns that searches for another event class, the price limit event.

The classes com.espertech.esper.example.stockticker.event.StockTick and PriceLimit represent our events. The event patterns are created by the class com.espertech.esper.example.stockticker.monitor.StockTickerMonitor.

Summary:

In the MatchMaker example every mobile user has an X and Y location, a set of properties (gender, hair color, age range) and a set of preferences (one for each property) to match. The task of the event patterns created by this example is to detect mobile users that are within proximity given a certain range, and for which the properties match preferences.

The event class representing mobile users is com.espertech.esper.example.matchmaker.event.MobileUserBean. The com.espertech.esper.example.matchmaker.monitor.MatchMakingMonitor class contains the patterns for detecing matches.

Summary:

This example handles very minimal temperature sensor events which are represented by java.util.Map. It creates a named window and fills it with a large number of events. It then executes a large number of pre-defined queries via on-select as well as performs a large number of fire-and-forget queries against the named window, and reports execution times.

Virtual data windows are an extension API used to integrate external stores and expose the data therein as a named window.

See the virtualdw folder for example code, compile and run scripts.

The example is also discussed in the section on extension APIs specifically the aggregation multi-function development. The example uses the jgrapht library for a cycle-detection problem detecting cycles in transactions between accounts.

See the examples/cycledetect folder for example code, compile and run scripts.

This example develops some code for measuring quality-of-service levels such as for a service-level agreement (SLA). A SLA is a contract between 2 parties that defines service constraints such as maximum latency for service operations or error rates.

The example measures and monitors operation latency and error counts per customer and operation. When one of our operations oversteps these constraints, we want to be alerted right away. Additionally, we would like to have some monitoring in place that checks the health of our service and provides some information on how the operations are used.

Some of the constraints we need to check are:

The com.espertech.esper.example.qos_sla.events.OperationMeasurement event class with its latency and status properties is the main event used for the SLA analysis. The other event LatencyLimit serves to set latency limits on the fly.

The com.espertech.esper.example.qos_sla.monitor.AverageLatencyMonitor creates an EPL statement that computes latency statistics per customer and operation for the last 100 events. The DynaLatencySpikeMonitor uses an event pattern to listen to spikes in latency with dynamically set limits. The ErrorRateMonitor uses the timer 'at' operator in an event pattern that wakes up periodically and polls the error rate within the last 10 minutes. The ServiceHealthMonitor simply alerts when 3 errors occur, and the SpikeAndErrorMonitor alerts when a fixed latency is overstepped or an error status is reported.

Summary:

This example was developed for the DEBS 2011 conference and demonstrates how scoring rules for a trivia game can be implemented in EPL.

The EPL module that implements all scoring rules is located in the etc folder in file trivia.epl. The EPL is all required to run the solution without any custom functions required.

The trivia geeks club rules (the requirements) are provided in the etc folder in file trivia_scoring_requirements.htm.

The implementation we provide tests the questions, answers and scoring according to the data provided in trivia_test_questions_small.htm and trivia_test_questions_large.htm.