Presenting Diagnostics

In debriefing the BoF and the responses people had to the work that Chas presented, one common theme became apparent to the group. It is difficult for people to initially understand the scope and direction of the project, but once they grasp its nature the need and potential become clear to them. There had been similar problems in an effort to present the Diagnostics project to some systems managers at CMU proposing the MS-MOM project as an alternative solution. After some time of working through the issues, the systems manager began to understand the differences and see the potential.

Nate suggested looking at similar diagnostic projects that had been undertaken by Internet2 and their evolution and marketing, such as LOOK, a set of LDAP directory monitoring tools built on top of ORCA. Chas thought a useful deliverable would be targetting a second set of slides more effective in conveying the type of project undertaken. This would include more concrete mock-ups of such components as the common event record which may better demonstrate the power and utility of the system.

Scope Refining

Another possibility proposed during the BoF was the ability to perform complex diagnostics that climbed the protocol stack, such as looking at network traffic coming from a source which has the signature of a virus and correlating that with events on a host such as a log of applied patches. However, focusing on application baselining seemed like a fair place to start.

An important way to scope the project is by having an adequately specific charter. The group wordsmithed the phrase "application service", which to Von implied things such as Shibboleth, while to Mark, it seemed that the application service is the server-side component of a client-server application, such as a calendar server, web server, IMAP server, etc. Chas offered to wrap the entire discussion up by changing the term to "application system."

A lot of the work that will have to be done to make these tools a reality will be on the part of individual applications. For example, the Shibboleth coders will have to rationalize the error messages produced and appropriately tag them. This process might involve looking at the types of reports that institutions with Shibboleth currently deployed encounter at the help desk. In Steven's experiences thus far, resolving many problems has required integration of the logs from both the origin and the target; and, usually, the only the information there is to start from is a student calling the help desk, saying, "it's broken."

Building Scenarios

The whole of debugging as it currently stands is focused on whoever plays the role of Middleware Technician, interacting with the user and various systems to try to tease out a problem. This is where the group envisions where diagnostic tools could best simplify a task, and also where the best set of scenarios for those tools could be culled.

There's a rough consensus on how these tools will be structured and work, but Chas has a justified concern that the devils are in the details. He wants to spell out in the near future exactly how this is all intended to work, then proceed into conversations about the trade-offs offered by alternative architectures. The universally recognized method of doing this is, of course, scenarios. This would be expanded in this process to include, in each person's opinion, ten questions that the diagnostic service should be equipped to answer.

[AI] The various participants on the call were drafted to each perform a detailed examination of how an application would interact with a specified draft diagnostic service. [AI] Chas offered to draft a set of documents that would help to both describe this service and give an example of how to run a scenario against it.

Action Items

1. Many people were asked to draft scenarios about how an application would interact with a diagnostic service.
2. Chas offered to draft documents that would describe a diagnostic service and give an example of how to run a scenario against it.