======================================================================= MESSAGE IDENTIFIER: ISO/IEC JTC1/SC7/Secretariat/1996-38 ======================================================================= ISO/IEC JTC1/SC7 N1481 1996-02-16 TITLE: LETTER BALLOT SUMMARY CD 14598-1.2 Software Prodcut Evaluation - Part 1 General Overview SOURCE: SC7 Secretariat WORK ITEM: 07.13.02.01 STATUS: Second CD Ballot REFERENCES: SC7 N1430 ACTION: Information and Appropriate Action. Francois Coallier ISO/IEC JTC1/SC7 Secretariat ******************************************************** ISO/IEC JTC1/SC7 N1481 LETTER BALLOT SUMMARY Project: 07.13.02.01 Subject: Software Prodcut Evaluation - Part 1, General Overview Reference: N 1430 Ballot: Second CD Ballot Circulation date: 1995 08 08 Closing date: 1995 11 22 Circulated to: P, O, L members Circulated by: SC7 Secretariat -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- P MEMBERS APPROVING (* = WITH COMMENTS) --------------------------------------- BRAZIL *, CANADA , CZECH REPUBLIC, DENMARK *, FINLAND, FRANCE, GERMANY, IRELAND, ITALY , JAPAN *, NETHERLANDS *, NORWAY, SWEDEN *, UK *, UKRAINE P MEMBERS DISAPPROVING (* = WITH COMMENTS) ------------------------------------------ USA * P MEMBERS ABSTAINING -------------------- P MEMBERS NOT VOTING -------------------- BELGIUM, HUNGARY, KOREA, MEXICO, ROMANIA, RUSSIAN FEDERATION, SINGAPORE, SOUTH AFRICA, SPAIN, THAILAND O MEMBERS AND LAISIONS (* = WITH COMMENTS) ------------------------------------------ LATE VOTES ---------- AUSTRALIA (Approve) ***************************************** COMMENTS ======== BRAZIL ------ Brazilian comments to General Overview 1. Comments Item: 4. Definitions Comments: To elucidate any possible doubt, we suggest the inclusion of the following definitions, presented in the Std 610.12-1990/IEEE: failure: The inability of a system or component to perform its required functions within specified performance requirements. Note: The fault tolerance discipline distinguishes between a human action (a mistake), its manifestation (a hardware or software fault), the result of the fault (a failure), and the amount by which the result is incorrect (the error). fault: (1) A defect in a hardware device or component; for example, a short circuit or broken wire. (2) An incorrect step, process, or data definition in a computer program. Note: This definition is used primarily by the fault tolerance discipline. In common usage, the terms "error" and "bug" are used to express this meaning. error: (1) The difference between a computed, observed, or measured value or condition and the true, specified, or theoretically correct value or condition. For example, a difference of 30 meters between a computed result and the correct result. (2) An incorrect step, process, or data definition. For example, an incorrect instruction in a computer program. (3) An incorrect result. For example, a computed result of 12 when the correct result is 10. (4) A human action that produces an incorrect result. For example, an incorrect action on the part of a programmer or operator. Note: While all four definitions are commonly used, one distinction assigns definition 1 to the word "error", definition 2 to the word "fault", definition 3 to the word "failure", and definition 4 to the word "mistake". Item: 4.10 attribute. Comments: Taking into account that we can also have an unmeasurable attribute, as stated in ISO/IEC 9126-2, we propose a new definition, according to "WD for terminology - N1054": Attribute: An observable property of an entity. item: 4.20 indicator. Comments: we propose a new definition: indicator: A measure that can be used to estimate or predict another measure. Item: 5.1 Structure of standards Place: figure 1 Comments: Replace 1: General Guide by 1: General Overview Shouldn't it exist an association between the box General Overview and the box Evaluation Modules ? Item: 5.4.3 Internal metrics Place: 1st line Comments: "Internal metrics produce measures that are used to represent the quality of the static representation of the software", we suggest: "Internal metrics produce measures obtained from the static representation of the software". Item: 5.4.3 Internal metrics Place: 2nd line Comments: "They measure internal attributes of the software..." (instead of requirements") Item: 8.1 Select metrics Place: 3rd paragraph - final Comments: "..., because the metrics from the user's view is crucial." shouldn't it be "..., because the metrics from the user's view are crucial." ? DENMARK ------- Page 6: Names of parts of 14598 are incorrect. Page 8: Definitions - 4.13 and 4.14: Since a measure is a number or a category it is not meaningful to apply these definitions. It should be much more relevant to define direct and indirect measurement methods. - 4.15 and 4.16: These definitions should define internal and external metrics in order to comply with the terminology of ISO 9126 part 2 and part 3. Page 14: Supply - The text in 7.14 should describe the use of evaluation from the point of view of a supplier. From the text it seems that the supplier is developing the software. Page 18: References to ISO 9241 and IEC 50 (191) should be deleted from the main text - the references could be given as informative examples. ITALY ----- CD 14598-1.2, Software Product Evaluation - Part 1 (Title: General Overview) N 1430 Approval with the following comments 1) General Comment as User of the document The document is useful, because it makes easier a global understanding of the evaluation processthe 2) Comments on Specific Points Section 4.2 Definition of implied needs (which is absent in ISO 8402:1994) is appreciated, it is useful. To have provided this definition is a good way for stressing the importance of implied needs, very often neglected in the projects. Section 7.2 The distinction between external and internal quality is useful, and methodologically relevant, in order to finalise better the activities on internal measures to the determination of the corresponding external measures (more understandable by users and managers) 3) Editorial or orthographical remarks Section 7.3 (the last two paragraphs) Two standards are mentioned, which are ISO 9241-11, and IEC 50 (191). The first is listed properly in Annex A; the second does not appear neither in Section 2, nor in Annex A, and it should be listed in one of the two places. Annex A. A.2 Bibliography: Titles and other information are missing for the entries by Azuma and McCall JAPAN ----- (Note G: General Comment, TM: Major Technical Commnet, Tm: Minor Technical Commnet, E: Editorial Comment ) JPN-1G) "End product" and "Final product" are used in the text, i.e. 7.1.1, but those meaning seems to be the same. If so, one of them should be used for the consitency. JPN-2E) 2.Normative reference Edition of ISO 8402 "(New)" should be replaced by issued year, "1994". JPN-3E) 2.Normative reference Document numbers of 9126 part one, two and three should be corrected as 9126-1, 9126-2, and 9126-3. JPN-4E) 2.Normative reference Title of 14598-3 should be changed to "Process for developers". JPN-5E) 2.Normative reference Title of 14598-4 should be changed to "Process for acquirers". JPN-6E) 2.Normative reference Title of 14598-5 should be changed to "Process for evaluators". JPN-7E) 4.Definition "Evaluation module" is used in the ISO/IEC 14598 series commonly, since it should be defined in ISO/IEC 14598-1. JPN-8E) 4.1 "[New ISO 8402]" should be replaced by "ISO 8402, 1994". JPN-9E) 4.4 "ISO/IEC 12207-1" should be replaced by "ISO/IEC 12207, 1995". JPN-10E) 4.8 & 4.10 "ISO 9126" in NOTE of 4.8 and "ISO 9126-1" in NOTE of 4.10 should be changed to "ISO/IEC 9126, 1991" or "ISO/IEC 9126-1 (New)". JPN-11E) 4.14 "Data element" is used in the ISO/IEC 14598 series commonly, since it should be defined in ISO/IEC 14598-1 in conjunction with the definition of indirect measure. JPN-12Tm) 4.16 External measure will be derived from not only the behaviour of system but also the influence. So, "or influence" should be added at the back of "behaviour". JPN-13E) 4.20 "Indicator" is defined differently in ISO/IEC 14598-1 and ISO/IEC 14598-3. They should be made consistent. JPN-14E) 5.1 ISO/IEC 9126-2 and -3 will become technical reports. So, the title of 5.1 should be changed to "Structure of standards and technical reports". JPN-15E) 5.1 For the same reason with JPN-13E, the first sentence should be changed as below; "The ISO/IEC 9126 series defines software quality characteristics and gives examples of the metrics." JPN-16E) 5.1 Considering the context of second sentence, first word of the sentence, "ISO 14598", should be replaced by "The ISO/IEC 14598 series". JPN-17E) 5.1 For the same reason with JPN-13E, "these standards" should be replaced by "these standards and technical reports". JPN-18E) 5.2.2 Purchasing is included in acquisition in ISO/IEC 12207. So, "(or buy)" should be removed. And also, the parentheses of "(or has been developed)" should be removed. JPN-19E) 5.3 The latter "standards" seems to be redundant. It may be removed. JPN-20E) 5.3.1 The title of 14598-2 should be changed to "Planning and management". JPN-21E) 5.4.3 Explanation of internal metrics should start with "ISO/IEC 9126-3 describes ....", considering consistency of style with 5.4.1 and 5.4.2. JPN-22Tm) 6. Evaluation Process Relationships between evaluation process shown here and life cycle process model of ISO/IEC 12207 should be clarified. Evaluation process may be used in some processes and activities of ISO/IEC 12207, such as acquisition process, validation process or system qualification testing activity. Also, the other parts of the ISO/IEC 14598 series should refer to the evaluation process shown here. JPN-23E) Figure 5 "System characteristics" and "software characteristics" of the left side rectangles should be replaced by "external characteristics and subcharacteristics" and "internal characteristics and subcharacteristics" each. JPN-24E) Figure 6 "External measures of computer system" in the second rectangle of left side should be replaced by "external measures of software". JPN-25Tm) Figure 6 The meaning of arrow from external measures of computer system to internal characteristics of software should be clarified. NETHERLANDS ----------- Comments on N1430, CD 14598-1.2 Software product evaluation Part 1: General overview Editorial comments: Page 5, Introduction, 1st line: it could be clarified what the "general requirements" will refer to. Page 9, Definitions: the definition of a "stated need" is missing, which is important to decide whether a need is stated or not; the degree of definition (e.g. measurability or usefulness for requirements specification) might be a determining factor for the predicate "stated". Page 9, Definitions, 4.24 validation: the definition of validation occurs twice. Page 18, 8.1 Select metrics: the sentence " Metrics used in the development process should be correlated to the user respective metrics, because the metrics from the user's view is crucial" should be "..... are crucial". Page 18, 8.1 Select metrics: the sentence "Examples of metrics specifying the rules for measurement ...." would need some clarification: are the rules for measurement specified by the examples or by the metrics, and is the verb "specify" correct here? Technical comments: Page 7, Definitions, NOTE with 4.1 quality: the conclusion may be that in any environment all needs have to be stated: all needs have to be made explicit to be taken into account in both development and evaluation; the notion of "implied needs" seems to be rather intangible. Page 12, 5.4.1 Quality Characteristics and Subcharacteristics: the quality characteristics being the human perspective of quality may be doubted: in practice these notions are NOT part of "the language" of IT users; e.g. "reliability" in practice is not directly debatable with users, and has to be discussed using "higher level" notions like "availability". Therefore, it might be more appropriate to consider also the characteristics to be at an intermediate level between the user view and the metric. Page 19, 8.1.2 Requirements for measurements: second hyphen: could the data be obtained from observation of opinions (e.g. statistical data of user opinions for usability measurement)? Page 20, 8.2 Establish rating levels for metrics, figure 8: the figure suggests that "minimally acceptable" level, not meeting requirements, could be satisfactory; this does not seem consistent with the adopted quality interpretation and seems to make this approach unfit in a certification context. SWEDEN ------ The untroduction of EXTERNAL and INTERNAL QUALITY seems too complicated. If required, it must be much more clarified. UK -- 1. The document has too many repetitive diagrams. Figs 1,2 and 3 need to be replaced by one single diagram. 2. Clause 5.1 Fig 1 Replace "1: General Guide" With "1: General overview" 3. Clause 4.17 The undefined term of "scale" is used in the definition of "measurement scale". 4. Clause 4.17 Absolute measurements scales should be defined. 5. Clause 4.21 Replace: "quantitative scale" With: "measurement scale" 6. Clause 7.3 Replace: first sentence With: "One approach to the evaluation of software is to use a quality model which breaks it down into different characteristics". USA --- GENERAL COMMENTS USA 1. The USNB wishes to compliment and thank SC7/WG6 for their dedicated work in producing these documents. It is felt that the current versions of the documents, save Planning and Management, have improved a great deal over previous versions. The Planning and Management document needs much more work. As a whole, the documents are beginning to come together better as a unit, but more work needs to be done to make them more consistent and cohesive. The weakest part in this respect are conflicting terminology and the need to make the series read as if produced by one person. It needs to be made clear which of the documents will become standards and which ones technical reports. General Comments on the Parts 1, 2, 3 and 6 USA 2. The documents were reviewed against the SC7/WG6 Project 7- 13 Product Requirements dated 24 July 1995, which in itself remains open for comment. The US draft requirements document for this project dated 10 March 1995 was also considered. USA 3. There is also some concern over the lack of experience with the definition of characteristics and their associated measures. The US encourages that these documents undergo field trials prior to passage. It is upheld that this approach is more expedient than current ones in getting standards to the market. Overall comments: USA 4. The collection of documents constitutes primarily a guide. Only part 1 contains any requirements for conformance, two short clauses that net out to say that "measures shall be objective, empirical and reproducible" and that "specifications (apparently the same as measures in this context) shall either be in terms of the characteristics in ISO/IEC 9126-1 or another identified quality model." USA 5. There's still more confusion about conformance. According to Part 1, Section 8.3, conformance to part 1 does not depend on conformance to Part 3. So, why does part 3 have a conformance clause and what is its significance? USA 6. The relationship to ISO 12207 is not adequately explained. Only a tiny paragraph in Part 1 (Section 7.1.1) addresses the relationship at all, and that only superficially. Especially since this document describes acquirer and developer processes, it must be more tightly connected to 12207, the standard describing the relationship between acquirers and suppliers. USA 7. The basic concern with this version of ISO/IEC 14598 is that it, for the most part, precludes looking at subjective qualitative issues when evaluating the quality of a system. Part 1 definition of terms and Part 3 introduction, discuss qualitative and subjective measures, but the rest of the standard only focuses on metrics. There is a lot of research material supporting the notion of using the combination of metrics and rigidly defined subjective qualitative measures to assess the quality of systems. There are many critical areas with respect to quality issues that do not resolve into metric measurements. Issues about readability and understanding as well as consistency, which are underlying concepts of maintainability and reliability, are easier to quantify using qualitative measures. The amount of effort needed to resolve this issue is not insurmountable. Three types of changes are required. First, the term "quantitative" needs to be added to the definition section of Part 1. This definition needs to explicitly state that quantitative measures include both metrics-based numeric information and categorized qualitative observations. Second, almost every place that the term "metrics" is used it should be replaced with the term measures. The processes and rigor outlined in the standard are equally applicable to metric-base measures and qualitative measures. Third, all the examples used in the standard, including Part 5, should be reworked to include examples of qualitative measures as well as the current metric-based examples. USA 8. The entire 14598 series needs to editted in a consist manner. Currently each document reflects that there are separate editors for each document. For example, each document should have the same Foreword. Comments on Individual Documents Part 1: General Overview General Comments USA 9 . This document started off as the General Guide. Based upon the current requirements drafts, this document contains general requirements for planning, implementing, managing, executing, and assessing a software product evaluation process. It also addressees product quality measurement activities. USA 10. Editorial - the diagrams need to be brought up to date in terms of nomenclature. USA 11. It is unclear how the internal and external metrics relate to evaluation modules and to each other. This needs to be defined. USA 12. The document focuses on product quality. It should state, however, that quality is not the only criterion for the listed purposes. USA 13. Internal quality characteristics can also be applied to a complete software product. They do not only apply to intermediate products. The quality requirements should drive the selected types of measures. USA 14. There is an undue emphasis on external measures of product quality. The developer, for example, must rely largely on internal quality measures to support development decisions. USA 15. The discussion concerning metrics rating levels is too subjective. This adds to the already subjective nature of the measurement - quality model, which is loosely defined. USA 16. Sections 9 and 10 are only minimally addressed. - Should a general overview document be considered definitive enough to be a standard? - ISO 9000 is not addressed or referenced. Specific Comments USA 17. Part 1, Section 4. If section 4 is to contain all defini- tions for all the parts, then delete the definitions from the other parts and add to this part 1. The definitions should be in alphabetical order. USA 18. Part 1, Section 5, Figure 1 and others. These figure show lines connecting various boxes. The meaning to be attributed to the lines is not explained. Do the lines denote inheritance, decomposition, precedence, or what? Furthermore, Figure 1 also contains a dashed line that, apparently, means something different; what does it mean? USA 19. Part 1, Section 5.1 asserts that the relationship among several standards is explained by Figure 1. As in comment USA 17, this claim is untrue. USA 20. Part 1, Figure 2. See comment USA 17. USA 21. Part 1, Section 7.1.3, third paragraph. The first two sentences seem to be on the subject of "implied needs" and (general) "requirements". Their relationship to the subject of quality is not explained at all. Either relate them better or get rid of them. USA 22. Part 1, Section 7.1.1. This section introduces the term "quality in use." the US National Body is unfamiliar with this term--is it jargon or some personal phrase of the editor's? Please either define the term or convert it to language that the average software engineer will recognize. Also, if the language following the first use of the term is meant as a definition, then move the definition to the definition section. USA 23. Figure 5 on page 15. Why the double arrow between needs and quality in use? The US National Body believes that only the left arrowhead is needed. Also, meaning of the dotted lines. USA 24. Part 1, Section 7.3. The first sentence claims that "to evaluate software, it is necessary to use a quality model." This is merely an assertion without foundation. If the assertion is a fundamental premise of the standard then either support it or label it as an assumption. If not, get rid of it. USA 25. Part 1, Section 7.3. The last two paragraphs cite ISO 9241 and IEC 50(191). Neither of these are included in the list of normative references, though. The latter one is not listed in the bibliography either. Is normative reference intended?