MW-E2ED Conference Call March 7, 2007
Chas DiFatta, Carnegie Mellon (chair)
Mark Poepping, Carnegie Mellon
Paul Hill, MIT
Steve Olshansky, Internet2
Dean Woodbeck, Internet2 (scribe)
**EDDY Email application progress**
Chas reported that the email application is about ready to be tested at Carnegie Mellon. All syslog messages are being normalized into an "email blob." The blob includes information on who the email is from, who it is addressed to, the hops from the MX machines, and man other attributes. This does not include the subject line, the text of the message or any attachments. The email blob is the first stage of an email flow record which initially is used to experiment with the data structures and the UI for email diagnostics. The user interface will be used by two people at CMU - an email system administrator and a help desk person.
Chas has seen interest from industry and from researchers on campus who are interested in email diagnostics and EDDY. He would like to do a demo, at some point, but needs to figure out how to address any potential privacy implications before doing so.
Others at CMU are starting to adopt EDDY. Those doing sensor work in the architecture and civil engineering programs, for example, need event correlation and orchestration, as do researchers interested in botnet and security work. There are at least five anonymzed real-time (each 8000 events/sec) in use at this time feeding security and network researchers with network flow diagnostic data.
Chas anticipates sometime in the near future releasing the EDDY diagnostic email application. After about a few months of working through any glitches, Chas will give a status on the email applications progress.
FYI, IBM has provided funding to expand research and standards effort around the EDDY technology. Organizationally, EDDY has moved under CMU's CyLab within the Cyber-center for Diagnostics, Information and Telemetry (CyDAT).
**EDDY Feature Status**
There have been some adjustments made with the TopNetE application. A user will be able to search for more than just "top talkers." Someone can now ask, for example, for the top talkers for email or the top talkers by subnet.
Chas has completed some experiments with JAXB, used for creating and editing XML and a core part of EDDY infrastructure. Using JAXB 1.6 has been slow thus only enable the processing 2,000-3,000 events per/sec. While EDDY has two XML parsers (the high speed can go in excess of 15,000 events/sec) JAXB 2.0 has become available; and using that has provided a 38 percent performance increase, or about 4,500 events/second. Using JAXB 2.0 on some new Sun machines has yielded a performance of 30,000 events per second while the native parser can process events at around 30,000 event/sec. JAXB will be included in the EDDY framework on the next release and there is work on next generation architecture to push the performance above 100,000 events/sec per agent.
Chas reported he hopes to have progress on the CER Factory by the Internet2 Member Meeting. The CER Factory is a way for a non-Java developer use the EDDY event framework. It accepts a CER template and puts the CER on the EDDY backplane.
**Next Generation CER and Backplane Feature Requirements**
Over the last month, Chas has been considering functionality to be included in version 2.0. Considerations include searching for open source software than may provide more functionality for building a backplane. He is currently putting together requirements, resources, and doing a survey and some experiments.
Mark reported giving a quick talk on EDDY at the recent CSI2 working group face2face meeting in Cambridge. The meeting focused mostly on an incident security reporting tool (RENOIR) and exchanging flows of darknet information.
Next call April 4, 2007.