The method was based on the use of indicative case studies, conducted across the SPICE technical regions. Feedback was collected via trials-specific questionnaires and forms (to gather data for testing the hypotheses) and observation reports, as well as the project-wide problem reporting process.
Phase 1 Trials were conducted in three of the four SPICE technical centre regions: Europe, Canada and Pacific Rim. None were conducted in USA. The trials achieved reasonable coverage of the SPICE model for process management in a good cross-section of application domains. The data used in the analysis was collected from a total of 35 trials, 20 of which were conducted in Europe, 1 in Canada and 14 in Pacific Rim. Complete sets of questionnaires, context data and rating forms were received from 28 trials. In addition 100 observation and 128 problems reports were raised in total from 24 trials.
The results from the trials have provided very useful feedback from the user community for the improvement of the SPICE products. A number of conclusions and recommendations have been drawn from the results and experiences of the trials, relating to both the proposed standard and the trials process. It should be noted, however, that there were some inconsistencies in the application of the data collection process across the trials and this has tempered some of the conclusions drawn.
The SPICE model for process management was generally found to be relevant and applicable. However, major problems exist which are expected to necessitate a fundamental review and rework of the model for process management.
Although, assessments were generally easy to conduct, the SPICE document set needs to provide more direction and the rating of processes needs to be reviewed. The results also indicate that attention should be given to reducing the time needed to undertake a SPICE assessment.
Although profiles generated using the proposed rating mechanism were generally considered to provide an accurate and fair representation of the degree to which an organizational unit implements software process best practice, there was some difficulty in rating process management activities.
Conclusions about the guidance on the construction, selection and use of assessment instruments and tools are more tentative. It was generally considered that the guidance is useful for developing an assessment instrument; that the right instrument design characteristics are included; that the requirements for conformance are easy to understand; and that the indicator set provides adequate coverage. The indicators (which are provided to guide an assessor in recognising adequacy of software engineering or process management activities) were generally considered clear and easy to understand. Although the trials results were generally supportive of this part of the proposed standard, some substantial problems (regarding the size and organization of the indicator set, its complexity and maintainability and the requirements for assessment instruments) were raised by a small number of trials participants: these merit further investigation.
In retrospect, insufficient planning was undertaken for the trials to ensure: that the right (and only essential) data was collected; that a consistent approach was adopted across all trials, particularly with respect to the completion and submission of data in a timely manner; that problems with data entry were minimized; and that the analyses tasks were consistent and focused. In addition, insufficient time was allowed for the conduct of trials, for the collection and entry of data into the database for analysis, and for the analysis and report.
undertake a fundamental review and rework, as necessary, of the model for process management in the proposed standard to address the problems identified in the Phase 1 Trials regarding the relationships between generic practices and processes, and the interpretation and organization of generic practices.The remaining recommendations relate to more local problems that are less fundamental to the proposed standard. However, it is recommend that they are considered and that consequent amendments to the SPICE document set are identified and documented prior to commencement of Phase 2 Trials.
A number of recommendations are made relating to the trials process,
mainly relating to the planning stage. It is imperative that they are heeded
to ensure that the same problems are not encountered during Phase
2, especially as Phase 2 is likely to be a more complex and larger exercise.
Despite the problems encountered during the Phase 1 Trials, their successful
completion is a major achievement and has demonstrated the feasibility
of empirical validation of a full international standard.
Back to Top