Middleware End-to-End Diagnostics
Fall 2006 Internet2 Member Meeting, Chicago, IL
Monday, December 4, 2006

Mark Poepping, CMU (chair)
Chas DiFatta, CMU (chair)
Dan McCarriar, CMU
Chris Misra, U. Mass
Richard Machida, U. Alaska
Mary Kratz, U. Michigan
Morgan Passiment, AAMC
Michael McGill, Internet2
Erskine Clinton, U. Kentucky
Josh Tegart, UC San Diego
Guy Van Den Bergh, TERENA
Ingrid Melve, UNINETT
Wichan Lertwipatrakul, UNINET
Anthony Espinoza, UTSA
Shaun Abshere, WiscNet
Dave Donnelly, Stanford U.
George Laskaris, NJEDge.Net
Mike Holcomb, ATP/U.Arizona
Warren Leung, UCLA
Bryan Rohling, ESSDACK
Tim Stoddard, UALR
Chin “Sam” Tseng, TANet2/TWAREN
Bart Walz, Florida A&M
John Stier, Stony Brook U.

*Session Abstract*
The BoF will discuss progress of the Internet2 End-to-End diagnostics toolkit effort named EDDY. The group will also share experiences using the tool kit to develop an end-to-end Email diagnostic application targeted at enterprise Email administrators and help desks. They will also discuss plans for a new push into the areas of environmental and security diagnostics.

The following discussion was based on a presentation posted at
<http://events.internet2.edu/2006/fall-mm/sessionDetails.cfm?session=3010&event=258>.

*Discussion* {Chas} and {Mark} shared a presentation discussing the current targets of diagnostics in this project – email, environment, security, network. EDDY provides a framework for logging activity information for later metrics.

They reviewed use cases of Middleware End-to-End Diagnostics, explaining how real campuses are using it today. EDDY is not a notifier in the sense of managing the tool itself, but it does use log information as a data source.

Questions that remain include how to bring sources together for constructing a useful diagnostic tool, and furthermore, from which sources? Which messages get added to the flow and how does that happen? New messages are put in a new data structure, but the exact build is not certain as it is unnecessary to see its every hop.

The common message data, including cross machine keys if needed, queueID and size of the message are elements modeled in a mail system. {Mark} spoke of the destination data, which is the time from source to delivery for the message, and also of the exploder, which changes the primary keys when it maps the mail in and mail out. For example if the help-desk breaks, it is then possible to retrieve how many calls were missed.

EDDY allows you to process and match the information you want to look at, without having to manually filter through the information. You are able to see why a message got hung up or why it was never resent. You are able to see through the network to compile events for a certain user. A CER – common event record – factory allows you to create a CER from a limited set of attributes.

Privacy concerns present several issues, as there is much personal data being passed. It might be an option to scrub certain fields, per policy set at the institution. Security diagnostics presents another opportunity e.g., to see when and how a mail system hacker breaks in, likely destroying the logs.

1st Order Flow events can have many vendors and the model fits all flows; it strips privacy issues and remaps the address so it cannot be back-mapped. 2nd Order Events take the aggregate and then look at a slice from the byte perspective – packet payload.

An attendee asked about the challenges in forming a data model, especially for one new to EDDY. {Mark and Chas} recommended that they first establish the use case and work backwards. Most products are not built with diagnostics in mind, so it requires finger-pointing to design a data model around that. For example, primitive diagnostics might say a system or application is up or down, but they cannot keep track of where mail goes. Start with the information you have and look then to areas in need.

Future development direction includes simplification of new event injection and standardization.

Additional EDDY information can be found on the following sites:
<http://middleware.internet2.edu/e2ed/>
<http://www.cmu.edu/eddy/>

There were no new action items arising from this discussion.