ppy/OntologySummit2013_s12_chat-transcript_unedited_20130404a.txt ------ Chat transcript from room: summit_20130404 2013-04-04 GMT-08:00 [PDT] ------ [9:16] PeterYim: Welcome to the = OntologySummit2013: Virtual Panel Session-12 - Thu 2013-04-04 = Summit Theme: Ontology Evaluation Across the Ontology Lifecycle Session Topic: Ontology Summit 2013: Synthesis-II * Session Co-chairs - Professor MichaelGruninger (U of Toronto, Canada) and Dr. MatthewWest (Information Junction, UK) Agenda: * Professor MichaelGruninger (U of Toronto, Canada) - "Thoughts on Ontology Summit 2013 and session intro" * Dr. MatthewWest (Information Junction, UK) - "Reflections on Ontology Summit 2013" * Dr. LeoObrst (MITRE) & Dr. SteveRay (CMU) - "Track-A: Intrinsic Aspects of Ontology Evaluation - Synthesis-2" * Mr. TerryLongstreth (Ind. Consultant) & Dr. ToddSchneider (Raytheon) - "Track-B: Extrinsic Aspects of Ontology Evaluation - Synthesis-2" * Dr. MatthewWest (Information Junction) & Mr. MikeBennett (EDM Council; Hypercube) - "Track-C: Building Ontologies to Meet Evaluation Criteria - Synthesis-2" * Dr. MichaelDenny (MITRE) & Mr. PeterYim (Ontolog; CIM3) - "Track-D: Software Environments for Evaluating Ontologies - Synthesis-2" * Dr. AmandaVizedom (Ind. Consultant) & Dr. FabianNeuhaus (NIST), moderators - Open Discussion on how the syntesized ideas may be represented in the Communique draft Logistics: * Refer to details on session page at: http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2013_03_28 * (if you haven't already done so) please click on "settings" (top center) and morph from "anonymous" to your RealName (in WikiWord format) * Mute control: *7 to un-mute ... *6 to mute * Can't find Skype Dial pad? ** for Windows Skype users: it may be under the "Call" dropdown menu as "Show Dial pad" ** for Linux Skype users: please note that the dial-pad is only available on v4.1 (or later or the earlier Skype versions 2.x,) if the dialpad button is not shown in the call window you need to press the "d" hotkey to enable it. * Note: ... it has come to our attention that our conference bridge provider is running into some problems with the "joinconference" skype connections. In case anyone gets in trouble, please try to call the phone numbers instead (e.g. from your phone, skype-out, google-voice, etc.) . == Proceedings: == . [9:23] anonymous morphed into CarmenChui [9:25] anonymous1 morphed into MichaelDenny [9:25] anonymous morphed into FrancescaQuattri [9:30] PeterYim: @FrancescaQuattri - did you just connect to the call? [9:31] FrancescaQuattri: yup [9:31] FrancescaQuattri: Hi Everybody [9:32] anonymous morphed into mary panahiazar [9:33] anonymous1 morphed into JulienCorman [9:33] PeterYim: Hello mary panahiazar, Welcome! [9:34] anonymous morphed into BobbinTeegarden [9:34] JoelBender: @Peter - online with Skype - no microphone [9:34] mary panahiazar: mary [at] knoesis.org [9:35] ToddSchneider: All, I have to leave at 14:00 EDT. [9:36] PeterYim: == MichaelGruninger opens the session ... see: the [ 0-Gruninger ] slides [9:37] List of members: AliHashemi, AmandaVizedom, BobbinTeegarden, CarmenChui, dougFoxvog, FabianNeuhaus, FrancescaQuattri, FrankLoebe, FranLightsom, JoelBender, JulienCorman, KenBaclawski, mary panahiazar, MatthewWest, MeganKatsumi, MichaelDenny, MichaelGruninger, PeterYim, SteveRay, TerryLongstreth, ToddSchneider, vnc2 [9:43] MichaelGruninger: Outcome hackathon HC05 http://ontolog.cim3.net/file/work/OntologySummit2013/Hackathon-Clinics/HC-05_Ontology-of-OntologyEvaluation/wip/HC-05_doc-snapshot_at-end-day-20130331/ [9:46] AmandaVizedom: Note about HC-05 outputs: This is snapshot of work at tend of the weekend sessions. Results are dispersed across a number of text and graphic files. Currently, several of us are working on consolidating the conceptual model in both graphical and English text forms, and making sure that we, as a group, agree that this captures what we developed. We are also drafting formal ontologies based on this, in OWL and Common Logic, but all should be considered first drafts, and current push is on the consolidated concept model. [9:46] SteveRay: With respect to conditions for ontology evaluation, we can talk about necessary conditions for evaluation, and possibly sufficient conditions for evaluation, with respect to various stages of development. [9:47] PeterYim: == MatthewWest presenting ... see: the [ 1-West ] slides [9:51] SteveRay: Interesting: Decision taking (UK) = Decision making (USA) [9:55] PeterYim: @Amanda, Ali, et al. - at the OntoIOp working group meeting yesterday, TillMossakowski and I were kicking around the idea of hacking up a demo (for the OntologySummit_Symposium), to evaluate two manually developed versions of the "Ontology of Ontology Evaluation" (a la HC-05 - in OWL and CLIF), and two machine-translated versions of those Ontologies (of Onto Eval) with Hets / DOL / OntoIOp (OWL->CLIF; CLIF->OWL) ... and run them through some of the tools featured during this summit ... it'll be fun! [9:56] anonymous morphed into LamarHenderson [9:58] AmandaVizedom: Cost reduction benefits, and sponsor's ROI in general, were brought into our HC-05 discussions this weekend, advocated especially by BobSmith. Figuring out how these fit into the high-level evaluation has been a challenge. MatthewWest's comments related to his slide 3 suggests to me that we began to model requirements and their large dependence on usage, and we began to model aspects of usage, and we began to model purpose as part of that, but under purpose we focused on delivered functionality. Matthews slide 3 highlights delivered benefits, at a higher level than specific functionalities. That, I think, we need to add explicitly. [9:59] PeterYim: == SteveRay presenting ... see: the [ A-Obrst-Ray ] slides [10:00] AmandaVizedom: @Peter: Excellent! I've been a bit dissatisfied that even with our follow-on commitments to create the formal ontologies, we haven't had a specific plan for evaluating them. And that's no good, from the practicing what we preach perspective. So, in addition to the fun of it, I think that is an excellent idea! [10:02] anonymous morphed into Lamar Henderson [10:05] dougFoxvog: Class vs. instance distinction being questionable arises if the ontology makes the two disjoint. If classes may be used as arguments to predicates (and metaclasses are allowed), then one need not make the narrowest classes into instances of their superclasses. [10:07] dougFoxvog: [Above is in response to discussion of Slide 2 of Track A.] [10:15] PeterYim: == ToddSchneider presenting ... see: the [ B-Schneider-Longstreth ] slides [10:15] TerryLongstreth: I'm listening, but as Todd says, having trouble with verbal communication [10:17] SteveRay: Disagree with Terry in calling Oops! a blackbox evaluation. It is specifically examining the contents of the ontology - opening up the box and looking for structural errors. [10:18] TerryLongstreth: That was Todd, but I think he was just illustrating the ambiguity of the dichotomy [10:18] MatthewWest: @Ray: I would expect intrinsic properties to become important (or not) in supporting higher level extrinsic requirements. So the key is to understand the way higher level requirements are supported by requirements for generally lower level, intrinsic properties. [10:19] AmandaVizedom: @Matthew +1 (independently of Steve's comments or Oops!). [10:20] SteveRay: @Matthew: I agree. Intrinsic evaluation alone has no value unless related to the ultimate system performance. [10:20] dougFoxvog: I agree with Steve. OOPS! ignores the *meaning* of the terms, but has access to all the statements in the ontology. Ignoring the meaning seems to be what Terry meant by "black box". [10:21] dougFoxvog: @Matthew, @Amanda: +1 [10:21] MichaelGruninger: @dougFoxvog: What do you mean by "ignoring the meaning"? The "meaning" of a term should be equivalent to the possible interpretations of the axioms [10:22] SteveRay: @Doug: You may be right in how Todd (sorry Terry, got the names swapped) intended to use the term black box, but that is an odd use of the term, somewhat opposite to what at least I understand it to mean. [10:23] dougFoxvog: The "meaning" of the term is defined for humans and humans use that meaning for labeling (e.g., [10:23] dougFoxvog: cells on a slide, info on medical records, etc. [10:25] dougFoxvog: @Michael: I agree that the meaning of an ontology in a vacuum is just the possible interpretations of the axioms. However, ontologies are (hopefully) used in conjunction with other systems, and so their mappings to those systems affects the meaning of the terms. [10:25] AmandaVizedom: @Todd: While discussing slide 3, you said that the evaluation has a context, and that when you know that context, then you can rank the results of your evaluation (metrics, etc). This sounds to me like a different framing, but in principle equivalent to a different process characterization that we have discussed. In this other characterization, The context comes first -- specifying the intended usage, gathering requirements. From this, evaluation criteria are identified that are relevant to answering whether these specific, context-driven requirements are satisfied, and evaluation is conducted over those criteria. Do you agree that both processes emphasize the contextuality of evaluation relevance equivalently? [10:26] SteveRay: @Michael: I'd be interested in your thoughts on the axioms when one is presented with, say, and OWL file that contains only sub/superclass relations and some allValuesFrom or someValuesFrom relations. In other words, no explicit axioms at all. [10:27] PeterYim: @Todd - (re. slide#7) I somewhat disagree that "testers are not familiar with ontologies" ... if we look at (and we should) test designers as among the "testers" (that's the group that's meaningful, we should not be talking about the test operators), then they simply do not qualify for the job if they are not familiar with ontologies [10:28] dougFoxvog: @Amanda: Should we expect the contexts to be defined (as you said they must be) using an ontology? I.e., are the context definitions to be stated in a formal logic using terms defined in an ontology? [10:28] MichaelGruninger: @dougFoxvog: In the work with MeganKatsumi, the intended meanings of terms are requirements that are formalized as intended models. We can then evaluate the ontology (using the axioms alone) to determine whether or not it meets those requirements i.e. whether or no there are intended models. When ontologies are used together, the intended models need to be in common. [10:29] PeterYim: == MatthewWest presenting ... see: the [ C-West-Bennett ] slides [10:32] MichaelGruninger: @Steve: I would say that subclass relations are still axioms. Of course, if these are all you have, then there will most likely be many possible interpretations of the ontology that do not correspond to the intended meanings. A great example of this is the relationship between OWL-S and SWSO. In cases such as this, I wonder what the requirements for the ontology are considered to be. [10:33] dougFoxvog: Slide 3: "The physical level would be an encoding in a formal language" such as OWL. This is an interesting definition of "physical". It would be nice for the slide to be edited to clarify this meaning. I might call this the "code" level. [10:34] AmandaVizedom: @doug, yes, though here I am using context as I think Todd meant it, not in all the possible ways I might otherwise be found using it. ;-) In the HC-05 model, we've been so far following along with the Ontology Usage characterization seeds laid down in the 2011 summit. That is, the formalized characterization of context consists partially in the explicit capture of various aspects of the usage (including things like application type, users, and so on), not yet nearly exhaustively captured. Priority is on such characeristics as we come to understand that they make a difference to what ontology features are needed. [10:34] ToddSchneider: Peter, I qualified 'tester' to be in the context of system integration testing (i.e., the end of the development phases and prior to deployment). [10:35] PeterYim: == MikeDenny presenting ... see: the [ D-Denny-Yim ] slides [10:37] PeterYim: @Todd - fair! [10:38] AmandaVizedom: @Matthew - during HC-05, we found your Conceptual/Logical/Physical stages, following DB usage someone, to make the most sense when mapped thusly: Conceptual: human-centric capture in one or more artifacts, could be textual, graphical, combined, rigorous but not formal. Logical: expressed in a formal ontology language. Physical: expressed in a serialization of such a language. Is this compatible with your thinking? [10:39] LeoObrst1: Finally joining. Sorry I'm late. [10:39] PeterYim: glad you made it, Leo! [10:44] ToddSchneider: Matthew, Instead of 'quality', would 'value' be a notion that better conveys our intent? [10:46] TerryLongstreth: Track D makes a good point that much of our work has seemed to presume a Waterfall model of development. We didn't explicitly talk about it but the Track B concerns with dynamics are probably best illustrated in current practice by environments by dynamic injection of new or unanticipated requirements as happens in aglie development situations. [10:46] TerryLongstreth: c/aglie/agile [10:47] MatthewWest: @Amanda: Possibly. In truth there are variations in interpretation of the levels in the database world. Certainly the physical level is what is in the system running queries. The logical level is an abstraction of that that is not implementation environment specific. I would probably want to say that you would not have committed to FOL or DL yet, but we could debate that (maybe another level?) [10:48] dougFoxvog: There have been several mentions that symmetric, reflexive, and transitive predicates should have the same domain and range. This is true for symmetric predicates, but for transitive predicates, the requirement should be that the range is a subclass of the domain. For reflexive predicates, it really depends upon one's definition of "reflexive" -- does it mean (forAll (X P P_Range P_Domain) (implies (and (isa P BinaryPredicate) (range P P_Range) (domain P P_Domain) (isa X P_Range) (isa X P_Domain)) (P X X))) or (forAll (X P P_Range P_Domain) (implies (and (isa P BinaryPredicate) (range P P_Range) (domain P P_Domain) (isa X (ClassUnionFunction P_Range P_Domain)) (P X X))). In the second case, the domain & range must be the same. In the first, they should just not be disjoint. [10:53] SteveRay requests a private chat with you [10:54] FabianNeuhaus: whoever is breathing into his microphone, please mute yourself :-) [10:54] SteveRay: +1 to that. [10:58] SteveRay: @MichaelDenny: Indeed, some of us are trying to link ontology evaluation to traditional modeling tools. I and my team convert Enterprise Architect files into OWL, and then apply various evaluation queries against them using SPARQL. [10:59] SteveRay: One example of output can be found at http://fsgim.sv.cmu.edu [11:00] ToddSchneider: Have to go. [11:01] PeterYim: == Q&A and Open Discussion on how all of these ideas should be captured into the OntologySummit2013_Communique ... moderated by FabianNeuhaus and AmandaVizedom [11:01] PeterYim: refer to communique outline at: http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2013_Communique/Draft#nid3O16 [11:03] MichaelDenny: @SteveRay Very intersting and glad to see it. I will take a look. [11:04] ToddSchneider: Amanda, Fabian, One suggestion before I really leave, I'd suggest dropping the in/extrinsic distinction and replace it with the lifecycle phase. It seems a better criteria for making evaluations distinctions. [11:04] PeterYim: +1 to Todd's suggestion [11:06] PeterYim: I think the intrinsic/extrinsic distinction served a useful purpose to help us frame the discourse, but introducing this "new terminology" is as confusing as not introducing it at all [11:06] AmandaVizedom: This is also of great potential use to Enterprise Architecture and Business Process Management practices themselves, and the development of semantic IT to better support them. Enterprise semantic tech projects are often based in information sharing needs related to business processes. In best cases, that basis is somewhat clear from documentation of business process and EA environment from just such tools. But these tools stop at the level of the input, output, or sharing of information bearing objects (reports, data sets, messages). They don't drill down into the information *contents*. That is precisely where the ontology coverage needs and scoping of the semantic projects picks up, and it is much more effectively captured and conveyed within a context of continuity with those EA/BP models. [11:06] TerryLongstreth: @Fabian - Track B wasn't so concerned with the physical level as the behavioral consequences to the system of having ontology or an ontology within it. [11:06] LeoObrst1: @MichaelDenny: yes, I call it "a label does not wear its semantics on its sleeve", which a lot of XML and database folks sometimes think, e.g., if a label is named "Person", well of course I know what it means! This is also encouraged by very long camelCase concept names like "PersonsWhoWieldHammers", where the label seems to be the composition of the semantics of natural language terms. [11:06] SteveRay: @Todd: Not sure I agree with this. Lifecycle has to do with WHEN, or at which phase, does one evaluate. The intrinsic/extrinsic distinction relates to WHAT one is evaluating. [11:09] MichaelDenny: @LeoObrst ...or "you can't tell a concept by its cover" [11:09] MeganKatsumi: @SteveRay, @Todd: I agree that the in/extrinsic distinction is confusing, but I also think that Steve has a point about the proposed using of the lifecycle phase. Might another useful distinction be the idea of functional/non-functional requirements/attributes? [11:10] TerryLongstreth: Lifecycle phases may also have multiple contxts: to the developer, the lifecycle phase labeled development is (one of) his operational swimming pools. He may touch more than one ontology if for example, the development environment is driven by an ontology (Rational products have that flavor, if not directly employing the term) [11:10] FabianNeuhaus: @Terry - yes, that's what I meant, I did not put it very elegantly. My point was that there are some aspects of ontology evaluation/quality that was not covered by either track, should be covered. [11:12] MichaelDenny: @MeganKatsumi I have suggested "model quality" vs "domain fidelity" vs "application fitness". [11:12] SteveRay: @Michael: I like your partitioning. [11:12] MatthewWest: I also agree that intrinsic/extrinsic has not been helpful. However, I don't think it matters very much. It gave us a way to start, and we can move on from that. [11:12] AmandaVizedom: As Fabian is saying on the conversation now, we do not plan on using the intrinsic/extrinsic distinction an organizer of the Communique. See outline. [11:12] anonymous morphed into PavithraKenjige [11:14] MichaelGruninger: Does it make sense to consider specific ontology evaluation tasks, and then specify what the inputs to the tasks are? e.g. is evaluation done with respect to the ontology's axioms alone? Is the ontology evaluated wrt a specific set of requirements? [11:14] PeterYim: +1 to what FabianNeuhaus just said about how he and AmandaVizedom are planning to lay out the communique [11:15] JackRing1: Seems to me any ontology must be evaluated with respect to domain-specific (usage) and discipline-specific (principles and standards) contexts. Further, an ontology can be evaluated for quality (what it is, what it does and what it knows), parsimony and beauty. I sense confusion about whether ontology serves as framework, praxis, system or what? [11:18] AmandaVizedom: I will add that I believe that there are many ways of "slicing and dicing" ontology characteristics/ requirements / evaluation criteria. Intrinsic/extrinsic is one (or several, given the various interpretations), as are lifecycle stages, relationship to aspects of usage/ relationship to some aspect of theory, etc.. And different tools and methodologies utilize different such organizations. What's more important is that we understand what the characteristics/criteria/requirements are, and when & why they matter, and how & when they may be evaluated. [11:18] MatthewWest: @Michael: You can only evaluate against requirements. If you look at my slide on Properties key to Information Quality, you will find properties at a level that business folk can state their requirements at. But then take consistency. What are the more detailed properties of an ontology that you can measure that tell you about its consistency? how do you transform the requirements at the business level down to this level? [11:19] MatthewWest: @Amanda: +1 [11:19] JackRing1: Life cycle is a distracting notion. Most all ontologies evolve and morph. It may be better to telk in terms of Usage Scenario. [11:19] PeterYim: Registration (either onsite or remote) is now open for the OntologySummit2013_Symposium at NIST - Thu & Fri May 2~3, 2013 (Gaithersburg, Maryland, USA) - see detials at: http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2013/WorkshopRegistration (registration for onsite attendance is mandatory ... so note the Apr-22 registration deadline!) [11:20] PeterYim: Join in the fun at this weekend's Hackathon-Clinics Activities - see details at: http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2013_Hackathon_Clinics#nid3PG4 ... even if you don't plan to hang around all day, you might be interested to participate at the "open webcast segment" of the two projects being featured this Saturday (Apr-6) [11:20] dougFoxvog: Leo is discussing properties of different ontology aspects relative to life cycle phase. If the specific relations are written down, they could be encoded using the ontology evaluation ontology. [11:20] MatthewWest: Sorry I have to go now. [11:20] PeterYim: Again, solicitation to software environment stewards and tool developers to respond to the OntologySummit2013 Software Survey - goto: http://ontolog-02.cim3.net/wiki/OntologySummit2013_Survey ... enter name of your tool, and proceed to questionnaire (make sure you complete all phases (questions under all tabs) [11:20] MichaelGruninger: @MatthewWest: Some of the criteria in Steve and Leo's slides use only the axioms of the ontology [11:20] PeterYim: As MichaelGruninger just said, same time next week, for OntologySummit2013 session-13: "Communique Draft Review" - Co-chairs: AmandaVizedom & FabianNeuhaus - developing session details at: http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2013_04_11 [11:21] MeganKatsumi: @Amanda: +1 [11:22] dougFoxvog: @Amanda: +2. The ontology evaluation ontology should have concepts and relations for all that. [11:25] SteveRay: Good session. Thanks! [11:25] PeterYim: great session! [11:25] PeterYim: -- session ended: 11:25 am PDT -- [11:25] List of attendees: AliHashemi, AmandaVizedom, BobbinTeegarden, CarmenChui, FabianNeuhaus, FranLightsom, FrancescaQuattri, FrankLoebe, JackRing, JackRing1, JoelBender, JulienCorman, KenBaclawski, Lamar Henderson, LamarHenderson, LeoObrst, LeoObrst1, MarcelaVegetti, MatthewWest, MeganKatsumi, MichaelDenny, MichaelGruninger, MikeRiben, MikeRiben1, PavithraKenjige, PeterYim, SteveRay, TerryLongstreth, TillMossakowski, ToddSchneider, anonymous, anonymous1, dougFoxvog, mary panahiazar, vnc2 [11:26] Sent transcript to: peter.yim@cim3.com ------