ppy/OntologySummit2013_s07_chat-transcript_edited_20130228b.txt url: http://ontolog.cim3.net/file/work/OntologySummit2013/2013-02-28_OntologySummit2013_OntologyEvaluation-ExtrinsicAspects-2/OntologySummit2013_s07_chat-transcript_edited_20130228b.txt ------ Chat transcript from room: summit_20130228 2013-02-28 GMT-08:00 [PST] ------ [8:53] PeterYim: Welcome to the = OntologySummit2013: Virtual Panel Session-07 - Thu 2013-02-28 = Summit Theme: Ontology Evaluation Across the Ontology Lifecycle * Summit Track Title: Track-B: Extrinsic Aspects of Ontology Evaluation Session Topic: Extrinsic Aspects of Ontology Evaluation - II * Session Co-chairs: Dr. ToddSchneider & Mr. TerryLongstreth Panelists / Briefings: * Dr. ToddSchneider (Raytheon) & Mr. TerryLongstreth (Ind. Consultant) - "Trinsics across the Engineering Lifecycle" * Dr. JoaoPauloAlmeida (Federal University of Espirito Santo, Brazil) - "Assessing Ontologies via Simulation" * Mr. KeithSilliman (Ind. Consultant) - "System Testing in a Data World" * Dr. AmandaVizedom (Ind. Consultant) - "From Business Requirements to Ontology Evaluation: A Process Outline" Logistics: * Refer to details on session page at: http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2013_02_28 * (if you haven't already done so) please click on "settings" (top center) and morph from "anonymous" to your RealName (in WikiWord format) * Mute control: *7 to un-mute ... *6 to mute * Can't find Skype Dial pad? ** for Windows Skype users: it's under the "Call" dropdown menu as "Show Dial pad" ** for Linux Skype users: please note that the dial-pad is only available on v4.1 (or later or the earlier Skype versions 2.x,) if the dialpad button is not shown in the call window you need to press the "d" hotkey to enable it. Attendees: AliHashemi, AmandaVizedom, AnatolyLevenchuk, AstridDuqueRamos, BobSmith, BobbinTeegarden, DavidLeal, DennisWisnosky, Diorbert, DougFoxvog, FranLightsom, FredHosea, GiancarloGuizzardi, HansPolzer, JoaoPauloAlmeida, JoelBender, JohnBilmanis, JohnGuerson, KenBaclawski, LamarHenderson, MeganKatsumi, MichaelGruninger, MichelleRaymond, MikeDean, OliverKutz, PeterYim, RamSriram, RichardMartin, SamirTartir, SimonSpero, SteveRay, TerryLongstreth, TiagoPrinceSales, ToddSchneider, TrishWhetzel, anonymous, anonymous1, vnc2 == Proceedings: == [9:18] anonymous morphed into TerryLongstreth [9:18] TerryLongstreth: KeithSilliman is with me on the same voice and chat connection [9:22] anonymous morphed into JoelBender [9:27] anonymous morphed into TiagoPrinceSales [9:27] anonymous1 morphed into DennisWisnosky [9:28] anonymous morphed into JohnGuerson [9:28] anonymous morphed into Diorbert [9:29] JoelBender: first time from Skype - but my PC has no microphone :-) [9:32] anonymous morphed into DougFoxvog [9:35] anonymous morphed into SteveRay [9:35] PeterYim: -- session started: 9:35am PST -- [9:35] PeterYim: == ToddSchneider opening the session on behalf of the so-chairs ... [9:35] JoaoPauloAlmeida: (referring to the way ToddSchneider pronounced his name) that's fine [9:37] PeterYim: == ToddSchneider presenting ... see: the [ 1-Schneider-Longstreth ] slides [9:39] anonymous morphed into AstridDuqueRamos [9:42] List of members: AliHashemi, AmandaVizedom, AstridDuqueRamos, BobSmith, BobbinTeegarden, DavidLeal, DennisWisnosky, Diorbert, DougFoxvog, FranLightsom, HansPolzer, JoaoPauloAlmeida, JoelBender, JohnBilmanis, JohnGuerson, MeganKatsumi, MichaelGruninger, MikeDean, OliverKutz, PeterYim, RamSriram, RichardMartin, TerryLongstreth, TiagoPrinceSales, ToddSchneider, TrishWhetzel, vnc2 [9:43] AmandaVizedom: @Todd: The shared version of the slides does not include the notes. Will you provide the references you mentioned to those lifecycle diagrams? [9:47] ToddSchneider: The 'Stack' model is called the GERA Life-Cycle Concept (Generic Enterprise Reference Architecture) from NIST. The 'V' diagram from http://www.mitre.org/work/systems_engineering/guide/se_lifecycle_building_blocks/ [9:47] AmandaVizedom: Thanks Todd! [9:43] PeterYim: == JoaoPauloAlmeida presenting ... see: the [ 2-Almeida ] slides [9:46] anonymous morphed into SamirTartir [9:47] anonymous morphed into GiancarloGuizzardi [9:55] HansPolzer: Dr. Almeida's talk is interesting. However, the scope of the model is implicit in whoever determines whether the model is under- or over-constrained. That's fine if the model is sponsored by a specific entity that can make those determinations, but it limits interoperability with those who do not share the perspectives of the sponsoring authority. I suppose someone else can try applying the model to their situation and on an experimental basis determine whether the model constraints are suitable to their situation [10:05] RichardMartin: Amanda - GERA is presented at: http://www.ict.griffith.edu.au/~bernus/taskforce/geram/versions/geram1-6-3/v1.6.3.html, and the Vee Model at: http://www.sebokwiki.org/1.0.1/index.php?title=System_Life_Cycle_Process_Models:_Vee [10:06] anonymous morphed into MichelleRaymond [10:07] PeterYim: Hi Michelle [10:10] anonymous morphed into AstridDuqueRamos [10:12] PeterYim: @JoaoPauloAlmeida & GiancarloGuizzardi - ref. our call for Proposals to the the "hackathon" and "clinics" activities - http://ontolog.cim3.net/forum/ontology-summit/2013-02/msg00056.html ... we would love to have your NEMO group participate; do consider sending in one or two proposals, please! [10:13] SimonSpero: @AmandaVizedom: I did a little experimentation with sampling to help locate errors in hierarchies [10:15] SimonSpero: @AmandaVizedom: Often there will be common nodes on inference paths that are more likely to in error [10:15] GiancarloGuizzardi: @PeterYim: thanks, Peter. We will surely consider that. [10:17] PeterYim: @GiancarloGuizzardi - great; thanks! [10:12] AmandaVizedom: @JoaoPauloAlmeida - regarding your scalability note on p. 27, have you considered extending this to do sampling & checking with larger scope (possibly even infinite) cases? Obviously, the checking would be less complete. But the challenges of verifying complex models against intended allowed states are such that even this partial, sample-based checking could be very valuable and informative, revealing many unintended implications of models. [10:16] SimonSpero: Sampling - Monte Carlo or bust [10:17] GiancarloGuizzardi: @SimonSpero & @AmandaVizedom: I would definitely like to see results on that direction. Simon, could you point me to some references of your work on that? [10:22] SimonSpero: GiancarloGuizzardi: nothing published, just running code. There was a visualization poster at DCMI (2008) that sort suggested the approach. Note intentional_with_a_t semantics of hierarchy are correct behavior http://dc2008.de/wp-content/uploads/2008/10/04_spero_poster.pdf [10:22] SimonSpero: @GiancarloGuizzardi: the errors present are incorrect even by those principles. [10:23] GiancarloGuizzardi: @SimonSpero: thanks a lot for the references [10:22] AmandaVizedom: Thank you @JoaoPauloAlmeida -- I spent some time in the past representing things such as (1) military terrain analysis (that is, analysis with respect to tactical implications) and (2) multi-stream situational data flows in which anomaly patterns can indicate probability of system or entity behavior. These models, built with domain SME input, are very hard to validate, and I can see how simulation-based validation would have been extremely valuable to those projects, even though they would have to be partial. Nice stuff. [10:25] JoaoPauloAlmeida: @AmandaVizedom: thanks, we have found the approach useful for modellers to gain confidence on what they wrote, and now are investigating visualization techniques to expose the simulation results to SMEs [10:18] PeterYim: == KeithSilliman presenting ... see: the [ 3-Silliman ] slides [10:23] anonymous morphed into FredHosea [10:44] anonymous morphed into LamarHenderson [10:35] TerryLongstreth: I inadvertently left the number off Slide 6 [10:35] HansPolzer: KeithSilliman's talk doesn't seem to be specific enough to ontology evaluation [10:36] SimonSpero: In many agile approaches, the test and behavior is a big part of the specification (e.g. http://en.wikipedia.org/wiki/Behavior-driven_development ; http://en.wikipedia.org/wiki/Test-driven_development ) [10:37] ToddSchneider: Hans, perhaps not, but he's brought to light other aspects of testing that haven't been brought up yet. I'll mention some after his talk. [10:41] BobbinTeegarden: @Todd: I agree, e.g. he addresses Reliability Availability Maintainability Security with are four of the RAMPSS criteria DEC placed over all of engineering, and are as germane to an ontology as a useable system. (the other two criteria were Performance and Scalability, also germane IMHO). [10:41] PeterYim: @HansPolzer - I agree with Todd - KeithSilliman *is* bringing to light aspects that haven't been brought up yet - if we consider the ulterior motive of getting application of ontologies ready for prime time, then these are *key* issues [10:43] AmandaVizedom: @HansPolzer - It may appear so, partly because Keith is specifically addressing *extrinsic* evaluation aspects ... by definition, aspects that are not intrinsically ontological. [10:45] SimonSpero: @Todd: Everybody tests everything. [10:46] SimonSpero: @Todd: QA are the people who aren't expected to fix the stuff they break so try much harder [10:46] SteveRay: I do think we have to be careful not to expand the scope into entire system testing. That's of course critical, but the summit is about Ontology Evaluation, which is a subset of testing of a system built on ontologies. [10:47] TerryLongstreth: (referring to the author of the book KeithSilliman was recommending) Weinberg (Jerry?) [10:47] JoaoPauloAlmeida: http://en.wikipedia.org/wiki/Gerald_Weinberg [10:51] HansPolzer: Terry, I believe the author is Gerald Weinberg. I have his book on "egoless programming" somewhere in my library. [10:55] TerryLongstreth: @Hans - thanks [11:02] TerryLongstreth: Keith's recommendation - Gerald Weinberg "Introduction to General Systems Thinking" [10:48] MichaelGruninger: @Todd: In your statement, you are mixing metaphor with methodology. When you talk about "reliability of an ontology", this seems metaphorical, since you want to use some notion of reliability from software engineering and try to figure out how it makes sense for an ontology. On the other hand, your other statements were dealing with methodology -- how can we determine when errors in software are actually errors in the ontology used with the software? [10:54] ToddSchneider: Michael, Possibly. I hadn't previously considered what it might mean to 'apply' the notion of reliability explicitly to an ontology apart from the system of which it's a part. However, if the use of an ontology engenders demonstrable actions (and the system can dynamically change the ontology), then notion of 'reliability' becomes pertinent. [11:02] HansPolzer: @Michael - one issue regarding reliability concerns "failure modes", and the features of a design that control failure modes or the way a system can respond to those failures. Translating that into the ontology domain isn't straightforward, but might include explicit representation of scope limits in the ontology itself. Thus software applications of the ontology could check whether they are attempting to use the ontology outside its intended scope. This is as opposed to using the ontology correctly, but in a context within which it may not produce reliably correct results. [11:05] MichaelGruninger: @HansPolzer: I just wanted to point out the distinction between some notion of "reliability of an ontology" on the one hand, and on the other hand, the role that reliability of a software system plays in the evaluation of the ontology. [11:01] ToddSchneider: I'm sorry but I have to attend another meeting. Thank you to the speakers for providing very useful material and ideas for this summit. Cheers. [10:48] PeterYim: == AmandaVizedom presenting ... see: the [ 4-Vizedom ] slides [11:07] PeterYim: @Amanda - I really like your process ... we need your involvement in the "Application Clinics" activities! [11:11] BobSmith: Good distinctions about role distinctions between the business people, the IT people, and the ontology people [11:14] TerryLongstreth: @Bob, I agree. I think Amanda has rather crisply laid out the intersection between ontology and enterprise system development [11:18] HansPolzer: @Terry, I agree. There is a lot of similarity/overlap with the SCOPE workshop process and stakeholder/participant selection process in the SCOPE Practitioner's Guide. [11:20] PeterYim: == Q&A and Open Discussion ... [11:25] TerryLongstreth: Ontology requirements are an outcome of early steps in system design; in other words, you have to decide first that you have evidence of the applicability of semantic technology ... if we're using Amanda's presentation as guidance, the decision to use an Ontology or any other semantic technology is her Page 15/Step 3 [11:29] HansPolzer: @Terry, I don't think that evidence of applicability of semantic technology is a good pre-condition. Most people don't see such evidence even while they fight over the meaning of data models. In the SCOPE workshop guide, we suggest the need for a "data architect", and often get dumb stares when we ask stakeholders whether they have one. And the data architect term is intended to include "ontologist", but we would typically get even stranger looks if we used that term with most stakeholders. [11:26] PeterYim: @ALL: I am soliciting help from everyone here: -- for software environment stewards and tool developers, please make sure you participate in the upcoming survey ---and help us get these colleagues of yours to respond to the survey too (they'll be on a wiki, so everyone will know who has or hasn't responded) ... or provide us with pointers so we can reach out to them -- we need ontology evaluation experts and tool developers to participate in the "hackathon" and "clinics" activities [11:27] PeterYim: Ref. our Call for Proposals to the the "hackathon" and "clinics" activities - http://ontolog.cim3.net/forum/ontology-summit/2013-02/msg00056.html - (to help tease out final issues before proposal submission) let's have a "hackathon" and "clinics"proposal preparation meeting next Tue 3/5 - can start at 7:00am PST - 7:00am PST / 10:00am EST / 4:00pm CET / 15:00 GMT/UTC - http://www.timeanddate.com/worldclock/fixedtime.html?month=3&day=5&year=2013&hour=7&min=0&sec=0&p1=224 - I will announce this on the [ontology-summit] list too (later) [11:33] JoaoPauloAlmeida: (referring to how Terry pronounced Amanda's first name) Your Brazilian pronunciation of "Amanda" is amazing. [11:33] AmandaVizedom: :-) [11:33] JoaoPauloAlmeida: :-) [11:33] PeterYim: join us again, same time next week (Thu 2013.03.07), for OntologySummit2013 session-08: "Intrinsic Aspects of Ontology Evaluation - II" - Co-chairs: LeoObrst & StevenRay - http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2013_03_07 [11:33] JoaoPauloAlmeida: Thank you all. [11:34] PeterYim: great session ... thanks everyone! [11:34] PeterYim: -- session ended: 11:34 am PST -- ------