ISO/IEC INTERNATIONAL STANDARD 14756 First edition 1999-11-15 Information technology Measurement and rating of performance of computer-based software systems Technologies de Iinformation Mesurage et gradation de la performance des systemes de logiciels d'ordinateurs IEC Reference number ISO/IEC 14756:1999(E) ISO/IEC 14756:1999(E) Contents Foreword. Introduction. VI Section 1: General 1 Scope. 2 Conformance 3 Normative reference.... 4 Definitions. 5 Abbreviations and symbols .. 5.1 Abbreviations .... .7 5.2 Symbols . .8 Section 2: Principles of measurement and rating.. 10 6 The measurement.. 10 6.1 Configuration requirements... 6.2User emulation.. 6.2.1 Random user behaviour.. 6.2.2Remoteterminalemulator... 10 6.2.3 Workload parameter set .. 6.2.4 Parameter set for proving the accuracy of the user emulation .. 11 6.3 The measurement procedure ... 12 6.3.1 The time phases of the measurement procedure .. 12 6.3.2 Writing a measurement logfile 6.3.3 Writing a computation result file... 13 6.4 Proof of validity of the measurement .... 6.4.1 Proof of the CBSS's computational correctness . 13 6.4.2 Proof of the remote terminal emulator's accuracy 13 6.4.3 Proof of the measurement result's statistical significance 13 7 Calculation of the performance values of the SUT 7.1 Mean execution time.. 7.2 Throughput ... 7.3 Timely throughput. 8 Basic data for rating 14 8.1 User requirements ... 14 8.2 The reference environment for rating software efficiency.. 8.2.1 Reference environment for assessing application software efficiency.... 8.2.2 Reference environment for assessing system software efficiency 15 ?ISO/IEC1999 All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying and microfilm, without permission in writing from the publisher. ISO/IEC Copyright Office·Case postale 56· CH-1211Geneve 20·Switzerland Printed in Switzerland ii @ ISO/IEC ISO/IEC 14756:1999(E) 9 Rating the performance values... 15 9.1 Computing the performance reference values . 9.1.1Meanexecutiontimereferencevalues.... 15 9.1.2 Throughput reference values... 15 9.2 Computing the performance rating values ... 15 9.2.1 The mean execution time rating values ... 15 9.2.2 Throughput rating values .... .15 9.2.3 The timeliness rating values . 9.3 Rating the overall performance of the SUT ... 9.4 Assessment of performance.... 9.4.1Thestepsofassessmentprocess 9.4.2 Weak reference environment ... 17 Section 3: Detailed procedure for measurement and rating 18 10 Input requirements ... 18 10.1 The SUT description.. 18 10.1.1 Specification of the hardware architecture and configuration ... 18 10.1.2 Specification of the system software configuration. 18 10.1.3 The application programs . 19 10.1.4 Additional software required for the measurement run. 19 10.1.5Thestoreddata... 19 10.1.6 Additional information for proof. 19 10.2 The workload parameter se... 19 10.2.1 The activity types . 19 10.2.2 Activity input variation... .20 10.2.3 The task types with timeliness function and task mode.... 10.2.4 The chain types and their frequencies ..... .21 10.2.5 Preparation times' mean values and their standard deviations .... 21 10.3 Input for measurement validation ... .22 10.3.1 Correct computation results... 22 10.3.2 Variation of input data and its resulting output... .22 10.3.3 Criteria for precision of working of the RTE .... 22 10.3.4 Criteria for statistical validity of results..... .22 11 The measurement.... .22 11.1 The measurement procedure 22 11.2 Individual rating interval. 23 12 Output from measurement procedure. 25 12.1 Measurement logfile ... .25 12.2 Computation result file... 25 13 Validation of measurements . 26 13.1 Validation of the computational correctness of the SUT 26 13.2 Validation of the accuracy of the RTE 26 13.2.1 Validity test by checking the relative chain frequencies . .26 13.2.2 Validity test by checking the preparation times... .