Heikki Uusitalo FiSMA r.y. ISO/IEC 29119 Part 1, Part 2, Part 3, Part 4 Testauksen prosessit, dokumentointi ja menetelmät Heikki Uusitalo FiSMA r.y.

Content ISO/IEC 29119 Testing standard Part 1 Concepts and Definitions Part 2 Testing Process Part 3 Test Documentation Part 4 Test Techniques

Tietotekniikan standardoinnista Kattavin standardointitaho tietotekniikan alueella on ISO eli International Organisation for Standardisation. Elektroniikan alueella standardoinnista huolehtii maailmanlaajuinen järjestö IEC eli International Electrotechnical Committee. Näillä on yhteinen standardointikomitea JTC1. Ohjelmistotuotannon ja järjestelmäkehityksen standardointi on tämän komitean alainen työryhmä SC7. Suomessa tietotekniikan standardoinnin yleisvastuu kuuluu ISO-jäsenyyttä hoitavalle Suomen Standardisoimisliitolle (SFS). Politiikkansa mukaisesti SFS on delegoinut SC7-standardoinnin käytännön työn FiSMA-yhteisölle (Finnish Software Measurement Association ry).

Business Planning Group JTC1 SC7 working groups (WG) SWG 1 Business Planning Group LCPHAG SC7 Life Cycle Process Harmonization Advisory Group Standards Management Group SWG 5 Vocabulary Maintenance SWG22 Secrétariat IT Enabled Services (BPO) WG27 Systems & Software Documentation WG2 Process Assessment WG10 Systems Quality Management WG23 CIF Usability WG28 Tools and Environment WG4 Techniques for Specifying IT Systems WG19 SLC Profiles and Guidelines for VSE WG24 IT Governance WG40 Software Product Measurement and Evaluation WG6 WG20 Software Engineering Body of Knowledge IT Service Management WG25 Life Cycle Management WG7 WG21 Software Asset Management WG26 Architecture WG42 Software Testing

Standardin synty ISO/IEC pitää vuosittain Plenary ja Interim –kokoukset, joissa työryhmien ehdotuksia käsitellään. Standardit valmistellaan työryhmissä ja hyväksytään äänestysmenettelyillä. SC7 –alakomitea työryhmä WG26 testaus. Suomen edustajana WG26 ryhmässä on TT Ossi Taipale LUT

Stages of Development: Standards Stabilized Standard A stabilized standard is one that has ongoing validity and effectiveness but is mature and insofar as can be determined will not require further maintenance of any sort. While a standard is in stabilized status it will no longer be subject to periodic maintenance but will be retained to provide for the continued viability of existing products or servicing of equipment that is expected to have a long working life. Stabilized Standard ~15-20 years Publication Stage 5 ISO/IEC ITTF Doc International Standard (ISO/IEC) 36 months Approval Stage 4 JTC 1 Doc Final Draft International Standard (FDIS) Draft International Standard (DIS) 33 months Committee Doc Committee Stage 3 Committee Drafts (CD) Final Committee Draft (FCD) 12 months Typical Timeframes Nominal schedule durations are the typical cumulative number of months following Approval of work item. Duration of stages may be shortened or lengthened depending on time To reach consensus to proceed to the next Stage.. Working Group Doc Preparatory Stage 2 Working Draft(s) (WD) Working Group Doc Proposal Stage 1 New Work Item Proposal (NP) New Work Item Proposal (NP) is a proposal for: a new standard a new part of an existing ISO standard revision of an existing ISO standard or part an amendment to an existing ISO standard or part a Technical Specification or a Publicly Available Specification Preliminary Stage 0 Preliminary Work Item (PWI)

FiSMA r.y. FiSMA r.y. Finnish Software Measurement Assoiciation M = Measurement, Management, Models 5 työryhmää ja foorumia: Standards and Models, Testing Models, IT Service Management, IT Effectiveness, SPIN FiSMAn jäsenistössä on nelisenkymmentä suomalaista ohjelmistotyötä tekevää ja palveluja ostavaa yritystä, sekä yliopistoja ja muita julkisen hallinnon organisaatioita. Jäseniä kiinnostaa standardien hyödyntäminen ja keskinäinen kokemusten vaihto jäsenyritysten välillä. Jäseniä mm. Trafi ja Endero.

Testauksen kaksi standardia Kaksi standardia, jotka täydentävät toisiaan ISO 29119 Testausprosessi ISO 33063 Testausprosessin arviointimalli ISO 29119 –standardi valmistunee tänä vuonna keskeisiltä osiltaan ISO 33063 –standardista on rakenne ja sisältö päätetty, työryhmä työstää sitä äänestyskuntoon. ISO 29119 on jo nyt siinä kunnossa, että sitä voidaan ja kannattaa hyödyntää Standardit täydentävät toisiaan, 29119 määrittelee prosessin ja 33063 perustuu siihen ja antaa perusteet arvioinnille.

Testauksen prosessit ja niiden kyvykkyyden arviointi

ISO 29119 Software Testing ISO 29119 Software Testing –standardin tarkoitus on määritellä ohjelmistotestaukselle kansainvälinen standardi, jota voidaan käyttää kaikissa organisaatioissa kaikenlaiseen ohjelmistotestaukseen. Standardi muodostuu neljästä osasta: 1. Concepts and Definitions Käsitteet ja määrittelyt 2. Test Process Testausprosessi 3. Test Documentation Dokumentointi 4. Test Techniques Testaustekniikat

Part 1 Concepts and Definitions Systems and Software Engineering Software Testing Part 1: Concepts and Definitions

Part 1 Concepts and Definitions Part 1 on informatiivinen osa, joka sisältää testauksen määrittelyt, käsitteet ja standardin sisällön. Part1 on myös johdatus standardin muihin osiin. Siinä kuvataan testauksen rooli organisaation ja projektin kannalta. Siinä kuvataan miten testaus sovitetaan erilaisiin ohjelmiston kehitysmallehin (agile, evolutionary, sequential ). Liitteissä kuvataan testauksen metriikkaa ja muita käytännön esimerkkejä. Määrittelyesimerkkinä test case, miten se ilmenee standardin eri osissa.

Part 1 Multi-layered text context Diagram

Part 1 Concepts and Definitions Esimerkki Test case a set of test inputs (and actions, where applicable), execution conditions, and expected results developed to exercise specific test coverage item(s).

Part 2 Test Process Systems and Software Engineering Software Testing

Part 2 Test Process Part 2 määrittelee yleisen testausprosessin, joka soveltuu kaikkien organisaatioiden käyttöön kaikessa ohjelmistojen testauksessa. Se sisältää testausprosessin prosessikaaviot ja -kuvaukset. Testausprosessi on kolmitasoinen: Organisaatiotason testausprosessi Testauksen hallintatason prosessi Dynaaminen testausprosessi Koska testauksen tarkoitus on riskien vähentäminen, niin standardin lähestymistapa on riskiperusteinen

Part 2 Test Process

Part 2 Test Process

Part 2 Organizational Test process Organizational test prosess -toiminto käsittää kaksi osaa: Testauspolitiikka Test Policy Testausstrategia Test Strategy Test Policy on lyhyt, jossa kuvataan testauksen tarkoitus, tavoitteet ja laajuus organisaatiossa. Se määrittää testauksen käytännöt, antaa puitteet testauspolitiikan ja strategian luonnille, katselmoinnille ja jatkuvalle kehitykselle. Test Strategy on yksityiskohtainen, tekninen dokumentti joka määrittelee miten organisaatio suorittaa testausta. Dokumentit ovat yleisiä, eivät projektikohtaisia

Part 2 Organizational Test Process implementation

Part 2 Test management process Muodostuu kolmesta aliprosessista Test Planning Process Tet Monitoring and Control Process Test Completion Process

Part 2 Test Management Process

Part 2 Test management process

Part 2 Dynamic Test Processes Dynaaminen testausprosessi kuvaa kuinka testaus suoritetaan testaustyypeittäin tai testausvaiheittain. Neljä prosessia Test design and implementation Test environment set-up and maintenance Test execution Test incident reporting

Part 2 Dynamic Test Processes

Part 2 Test design and implementation process

Part 2 Test Design & Implementation Process The Test Design & Implementation Process describes how test cases and test procedures are derived; these are normally documented in a test specification, but may be immediately executed, for instance, if performing exploratory testing, in which case they are unlikely to be documented in advance. In figure 10 the activities are shown in a logical sequence, but in practice iteration will occur between many of the activities, often with activities TD3 to TD5 occurring in parallel for substantial periods. 7.2.1 Purpose 7.2.2 Outcomes 7.2.3 Activities and tasks 7.2.3.1 Identify Feature Sets (TD1) 7.2.3.2 Derive Test Conditions (TD2) 7.2.3.3 Derive Test Coverage Items (TD3) 7.2.3.4 Derive Test Cases (TD4) 7.2.3.5 Assemble Test Sets (TD5) 7.2.3.6 Derive Test Procedures (TD6)

Part 2 Esimerkki 7.2.3.4 Derive Test Cases (TD4) This activity consists of the following tasks: a) One or more test cases shall be derived by determining pre-conditions, selecting input values and, where necessary, actions to exercise the selected test coverage items, and by determining the corresponding expected results. When deriving the test cases be aware that one test case may exercise more than one test coverage item and thus there is the opportunity to optimize derived test cases by combining coverage of multiple test coverage items in a single test case. b) Where appropriate, the test cases should be recorded in the test case specification. If recorded, the traceability between the test basis, feature sets, test conditions, test coverage items and test cases shall be explicitly described. d) Where appropriate, the contents of the test case specification should be approved by the stakeholders. This may require repeating tasks a) and b), and in some case first repeating the Derive Test Conditions (TD2) and/or Derive Test Coverage Items (TD3) activities.

Part 3 Test Documentation Systems and Software Engineering Software Testing Part 3: Test Documentation

Part 3 Test Documentation Standardi kattaa kaikenlaisen ohjelmistotestauksen dokumentoinnin mallit. Siinä määritellään dokumenttien käyttö ja sisältö. Dokumentaatio noudattaa Part 2 mukaista tasomallia: Organisaatiotason dokumentaatio Hallintatason dokumentaatio Testaustason dokumentaatio

Part 3 Test Documentation 7.3.4 Test cases Defines the test cases derived from the test coverage items. A test case specifies how one or more test coverage item(s) are exercised to determine whether or not that part of the test item has been implemented correctly. This section in the Test Case Specification could be formatted to list test cases under corresponding feature sets and/or test conditions. The test cases may be described in lists or in tables in a document or using a tool, e.g. A relational database or a dedicated test tool. The information for a test case includes: 7.3.4.1 Unique identifier Describes the unique identifier needed by each test case so that it can be distinguished from all other test cases. An automated tool may control the generation of the identifiers. 7.3.4.2 Objective Identifies and briefly describes the special focus or objective for the test case. This is typically in the form of a title. 7.3.4.3 Priority Defines the priority for the testing of this particular test case, if needed. 7.3.4.4 Traceability Describes traceability to the test coverage item that the test case exercises or lists reference(s) to the associated requirement(s) and/or design description(s) in the test basis. This may be documented in a Test Traceability Matrix.

Part 3 Test Documentation 7.3.4.5 Preconditions Describes the required state of the test environment and any special constraints pertaining to the execution of the test case, e.g. the state the test item shall be in before execution may start, including existence of specific test data and the currently active form or screen. This could be described explicitly or it could include references to other test cases, whose execution will set the preconditions. The environment needed may be described collectively for one or more feature sets, or it may not be described in this specification if the description in the Test Plan is sufficient. 7.3.4.6 Inputs Specifies each action required to bring the test item into a state where the expected result can be compared to the actual result. This may require provision of input data and events, e.g. button clicks, to the test item. Some of the input data may be specified by value, while others, such as constant tables or transaction files, may be specified by name. All appropriate databases, files, terminal messages, memory resident areas, and values passed by the operating system must be considered. All required relationships between input events, e.g. timing, must be described. The actions or steps may be numbered within the test case, if needed.

Part 3 Test Documentation 7.3.4.7 Expected results Specifies the expected outputs and behaviour (e.g. response time) that is required of the test item in response to the inputs. Provides the expected value(s) (with tolerances where appropriate) for each required output. The actions required to compare the expected result to the actual result may also be specified. For instance, examining the output in a field on a form that is not active when the input is provided, waiting for a batch job to run and a report to be printed out and examined, or closing down the test item and restarting it to examine stored data. 7.3.4.8 Test outcome and test result The description of a test case may include placeholders to record test outcome and/or test results during execution of the test case. Alternatively, this may be recorded in the Test Procedure Specification (see clause 7.10).

Part 4 Test techniques Testaustekniikoiden kuvaus Testauksen mittaamisen tekniikoiden kuvaus

Part 4 Test techniques, luokittelu

Part 4 Test techniques, esimerkki 5.3.1.3 Derive Test Cases (TD4) To derive test cases for statement testing the following steps should be followed: 1. Identify the test coverage items (the executable statements) in the test item (program). 2. Decide on the required level of statement coverage to be achieved. 3. Idntify ’new’ excecutable statements (statements that have not yet been exercised by the testing) in the program (it is normally best to start with statements towards the end of the program). 4. Identify the control flow subpaths that reach the executable statement from the start of the program. 5. Select the controlflow subpath that will exercise the most ‘new’ executable statements. 6. Determine the test inputs that will allow the control flow subpath to be exercised. 7. Determine the expected result that corresponds to the test inputs by applying the test inputs to the test basis. 8. Repeat from step 3 until the required level of statement coverage is achieved

Part 4 Test techniques, esimerkki 5.3 Structure-Based Testing Techniques 5.3.1 Statement Testing 5.3.1.1 Derive Test Conditions (TD2) A model of the source code of the test item (program) which identifies statements as either executable or nonexecutable shall be derived. Each executable statement shall be a test condition. 5.3.1.2 Derive Test Coverage Items (TD3) The executable statements shall be identified as test coverage items (i.e. they are the same as the test conditions).

Part 4 Test techniques, esimerkki 5.3.1.3 Derive Test Cases (TD4) The following steps shall be followed: 1. Identify control flow subpaths that reach one or more test coverage items (executable statements) that have not yet been executed during testing. 2. Determine the test inputs that will cause the control flow subpath(s) to be exercised. 3. Determine the expected outcome from exercising the control flow subpath(s) by applying the corresponding test inputs to the test basis. 4. Repeat steps 1 to 3 until the required level of test coverage of the executable statements is achieved.

Part 4 Test techniques, Testing of Quality Characteristics A.1 Quality Characteristics A.1.1 Overview Software testing can be carried out to collect evidence that required quality criteria have been satisfied by a test item. Required quality characteristics should be specified in the test basis. Definitions of quality characteristics could be derived from ISO/IEC 25010 System and Software Product Quality Requirements and Evaluation (SQuaRE) – System and Software Quality Models.

ISO/IEC 25010 Product Quality Model

Part 4 Test Techniques A.2 Mapping Quality Characteristics to Test Design Techniques The test design techniques described in this standard can be used to test a variety of the qualitycharacteristics listed above in Figure 3. The following table provides a mapping between them. Table 1 – Mapping of ISO/IEC 25010 product quality characteristics to test design techniques Statement Testing Functional Suitability Functional completeness Functional correctness Functional appropriateness

Further information about SC7 standards www.sfs.fi, IT standardisation at general level www.fisma.fi, Software, Systems, Services standards FiSMA: risto.nevalainen ( at ) fisma.fi