Discussion

Survey Processing

Chas reported that Ryan Muldoon has been making impressive progress in aggregating survey results developed for the diagnostic procedures of various applications. Different infrastructures that have been externally developed that may aid the diagnostic project have been examined, including Brian Tierney's NetLogger and AirCERT from CERT. Chas encourages participants to review the survey results located in the private section of the diagnostics web space frequently, as they're updated "roughly every week or so."

The immediate step Chas envisions will be to write a synopsis of what has resulted from this process. From here, the concept of the metatag can be better evaluated, and some "very simple code" can be written to attach metatags to logs and message flows. This will help gauge whether the tagging mechanism is sufficient to address various scenarios that have been proposed; Chas pointed out that Ken Klingenstein of Colorado believes this is a good approach.

Shibboleth Diagnostics

When asked whether the diagnostic question had come up at all in deployment thus far, Steven answered, "it doesn't get asked that way." This may be due to the current phase of deployment, and with the upcoming release of Shibboleth v1.2, there are likely to be many more production-level deployments. This will come with a concurrent decrease in the familiarity of deployers with the Shibboleth architecture and middleware in general.

This progress means diagnostics are firmly on the radar at this point, although it wasn't clear what needed the most attention. Chas' questions about whether installation, security, etc. needed to be addressed, led to an inconclusive response. Installation, Steven noted, "is a one-time issue. Lots of people get hung up there, or on the PKI issues."

The new version is much more flexible in its PKI support, which will both ease deployment and increase the potential for misconfiguration to cause failures. Benchmarking is more of an issue, due to the speed of XML signing done in Java: as fast as "a turtle with three legs cut off." By version 1.3, however, this signing will be done entirely externally to the Java code that makes up most of the origin.

Steven suspects that the biggest diagnosis problem will be that the support model will move from the developers addressing questions themselves to a help desk crew needing to handle failures. OCLC recently moved their Shibboleth target to production, but are reluctant to enable too many schools due to insufficient auditing capabilities.

Chas was curious about Shibboleth having problems with other layers in the protocol stack, such as the network. Steven was certain issues were inevitable, although there have been no reports of problems to date. This elicited more concerns from the group about the difficulty of recording such different types of events with a common format. Russ was worried that this would prove hard, and could end up limiting the scope of diagnostics in individual layers. The core question when considering such different systems may be, he pondered, "what is it that they have in common that allows you to define events sensibly when combined?"