Computer Validation. Risk analysis and system classification
Here you will find answers to the following questions: Which criteria can be used to classify computer systems into risk classes? Which are the GAMP classifications? Which measures should be taken depending risk class? |
Chapter 10 of the GMP Guide (see chapter 10 Risk Management) provides an excellent description of risk management in general. This chapter is therefore only intended to cover the practice-related, specific aspects of risk management for computerised systems.
9.D.1 GAMP classification
In the annexes of the GAMP‚4 Guide, software is divided into five classes and hardware into two classes, depending on their complexity. The classes are divided according to the validation requirement and deliverables.
The complexity and validation requirements increase from class 1 to class 5.
9.D.1.1 Class 1 - Operating system
In this class, only the name and the version are documented. The operating system is implicitly tested and hence covers the validation of the application, since all higher functions rely on this functioning flawlessly.
9.D.1.2 Class 2 - Firmware
Firmware is typically used in controls and instruments. Firmware can require that certain parameters are entered and configured, such as the date format, current date, time, etc. Firmware may also belong class 5, if it has been specifically programmed for an installation.
GAMP‚4, Annex M4 (Ed. ISPE) proposes recording of the name and version for firmware. This name and the version may not be visible from externally. In this case, it is pragmatic to record the equipment using design and serial number, since only externally accessible information is easy to verify and to test. Examples in this class include washing machines, drying ovens, balances, barcode scanners, etc.
The correct functioning of the software is tested and documented as a part of the installation qualification (IQ) and operational qualification (OQ). The configuration parameters must be verified and documented during qualification. It is important that all changes in the firmware, as well as any changes to the configuration parameters, are subject to change control. A detailed procedure for qualification is provided in chapter 6 Qualification.
Before start-up, it must be ensured that the necessary operating instructions (SOPs - Standard Operating Procedures) are available and that users are trained in these.
9.D.1.3 Class 3 - Standard software packages
Commercially available software normally falls into class 3. This typically includes analysis software in spectrophotometers, HPLC systems, gas chromatographs (GC), near infrared spectrophotometers (NIRA) and other laboratory instruments, as well as statistical evaluation packages. This class also includes all off-the-shelf software: COTS (commercial off the shelf), such as office programs.
The procedure for class 3 is the same as class 2. In addition, all calculation methods, algorithms, alarm messages and warning messages used must be checked and documented during qualification, normally operational qualification. For standard software packages used in highly-sensitive GMP areas with a high risk for product quality, it may necessary to execute a supplier assessment or even a supplier audit.
9.D.1.4 Class 4 - Configurable standard software packages
These are usually more complex applications that are available with a wide variety of standard interfaces. Typical members of this class are: MES (Manufacturing Execution Systems), ERP (Enterprise Resource Planning), control systems (SCADA - Supervisory Control And Data Acquisition) and DCS (Distributed Control Systems) such as room temperature controllers.
The complete functionality can be changed by configuration. Specific functions may often be realised using programs or macros; these should then be treated as class 5. Validation should be executed as described in chapter 9.E Validation of computerised systems and all details must be recorded as described above for class 1-3. In validation, particular attention should be paid to the configuration.
9.D.1.5 Class 5 - Customer-specific software
Whenever an application is programed for an individual application, it belongs to this class. Depending on the risk classification, in this case the whole set of validation activities must be executed. The details of documentation requirements are described in chapter 15 Documentation.
9.D.1.6 Validation tasks depending on classification
Listing the requirements that are detailed in the GAMP‚ Guide results in a table similar to the following, which contains a list of the validation tasks depending on system complexity (figure 9.D-1).
Firmware |
Standard |
Configurable systems |
Configurable |
Customer- |
||
---|---|---|---|---|---|---|
Inventory |
Classification |
X |
X |
X |
X |
X |
Risk evaluation |
X |
X |
X |
X |
X |
|
Inventory |
X |
X |
X |
X |
X |
|
Validation |
Validation plan |
X |
X |
X |
X |
|
Validation report |
X |
X |
X |
X |
||
Validation registry |
X |
X |
X |
X |
||
Qualification plan |
X |
|||||
Qualification report |
X |
|||||
Supplier management |
Supplier assessment |
X |
X |
X |
X |
|
Specify & Design |
User requirement specifications |
X |
X |
X |
X |
X |
System specifications |
X |
X |
X |
|||
Development standards |
X |
X |
||||
Project change control SOP |
X |
X |
X |
|||
Functional risk evaluation |
X |
X |
X |
|||
Technical design |
X |
X |
X |
|||
Build |
Source code, configuration, adjustment |
X |
X |
|||
Technical documents for IT |
X |
X |
X |
X |
||
Test |
Test protocols |
X |
X |
X |
X |
|
Unit/integration test specifications |
X |
X |
||||
Unit/integration test report |
X |
X |
||||
Installation test documentation |
X |
X |
X |
X |
X |
|
System test documentation |
X |
X |
X |
|||
Acceptance test documentation |
X |
X |
X |
X |
X |
|
Traceability matrix |
X |
X |
X |
X |
X |
|
System configuration basis |
X |
X |
X |
X |
X |
|
Implement & Use |
Implementation plan |
X |
X |
X |
||
User SOP |
X |
X |
X |
X |
X |
|
Change control SOP |
X |
X |
X |
X |
X |
|
Security administration SOP |
X |
X |
X |
X |
||
SOP for backup and restore |
X |
X |
X |
X |
||
SOP for archiving |
X |
X |
X |
X |
||
SOP for user training |
X |
X |
X |
X |
X |
|
Service level agreement |
X |
X |
X |
|||
SOP for periodic review |
X |
X |
X |
X |
X |
|
Plan for system shutdown |
X |
X |
X |
X |
||
System shutdown report |
X |
X |
X |
X |
9.D.1.7 Risk classification in accordance with GAMP‚
The GAMP‚ Guide contains an annex for risk classification that describs a simple and practically executable double classification of software.
Firstly, a risk is determined by evaluating the extent of damage and the frequency of an event. In the second stage, the probability of detection of an error is also considered, in order to determine the risk index (figure 9.D-2).
![]() |
9.D.2 Risk indexes
Scientifically, the risk is the product of the probability of occurrence and the degree of severity of an unwanted effect.
Risk = Probability of occurrence x Degree of damage x Probability of detection
Since it is difficult to work with abstract probabilities, in everyday life risk indexes are used, which are a combination of the probability of occurrence, the degree of damage, and the probability of detection, . It is common to determine the risk indexes empirically. An empirical method of this type is presented here. It is assumed that the risk index can be determined as a weighted sum of the different individual risks (see figure 9.D-3). For more details on risk analysis, see chapter 10.F Failure Mode Effects Analysis (FMEA).
9.D.2.1 Determining the risk index
The following diagram (figure 9.D-3) shows a possible example for determining the risk key figure.
Label |
Category |
Question |
Points |
|
---|---|---|---|---|
G |
Good Practice |
Can GCP, GLP, GMP or other Good Practices (SOX - Sarbanes Oxley) be applied? |
yes |
1 |
no |
0 |
|||
A |
Regulatory requirements at the implementation location |
Research |
1 |
|
Development - GLP (Good Laboratory Practice) |
3 |
|||
Development - GMP (Good Manufacturing Practice) |
5 |
|||
Development - GCP (Good Clinical Practice) |
5 |
|||
API manufacturing |
3 |
|||
Drug product manufacturing - GMP |
5 |
|||
Distribution and storage |
3 |
|||
Planning and logistics |
1 |
|||
Support processes - human resources |
1 |
|||
Support processes- information technology |
3 |
|||
Support processes - finances |
1 |
|||
N |
No. of users |
Although the probability of detection theoretically increases, the probability of occurrence and extent of damage also rapidly increase. 1 |
1 |
|
2 to 20 |
3 |
|||
21 to 100 |
7 |
|||
More than 100 |
10 |
|||
F |
Frequency of use |
Annually |
1 |
|
Monthly |
3 |
|||
Daily |
5 |
|||
V |
Geographical distribution |
At one workplace |
1 |
|
In one department |
3 |
|||
At one manufacturing site |
5 |
|||
Global |
10 |
|||
M |
Manual alternative solution |
Without great effort |
1 |
|
With extra effort |
3 |
|||
Only with considerable extra effort |
5 |
|||
Not feasible |
7 |
|||
P |
Importance of the data in the system |
Is the data only available in the affected system? |
yes |
1 |
no |
0 |
|||
I |
Inspection risk |
yes |
1 |
|
no |
0 |
|||
P |
Potential risk to patients |
No influence on patients |
1 |
|
Insignificant |
2 |
|||
Injury |
5 |
|||
Death |
10 |
|||
n |
Subsequent processes |
Has a correctable system failure been discovered in subsequent processes? |
yes |
1 |
no |
10 |
|||
S |
System category |
a - Firmware |
1 |
|
b - Standard software package |
3 |
|||
c - Configurable software package |
5 |
|||
d- c With custom-developed supplements |
8 |
|||
e - custom-developed solution |
10 |
|||
M |
Maturity or degree of maturity of the technology |
New technology |
10 |
|
Mature technology without internal company experience |
5 |
|||
Mature technology with internal company experience |
2 |
|||
Obsolete technology |
5 |
|||
o |
Openness, Internet |
Is part or all of the whole communication processed via the public Internet? |
yes |
10 |
no |
1 |
|||
RI = G * (A+N+H+V+M+W+I+P+N+S+M+O) |
Example:
A spreadsheet for the calculation of content uniformity determination in accordance with the pharmacopoeia in release analytics, which does not contain any macros, would have an RI of 42:
G |
Regulations applicable |
yes |
1 |
A |
Regulatory requirements at the implementation location |
GMP |
5 |
N |
No. of users |
2 to 20 |
3 |
F |
Frequency of use |
Daily |
5 |
V |
Geographical distribution |
In one department |
3 |
M |
Manual alternative solution |
With extra effort |
3 |
P |
Importance of the data in the system |
no |
0 |
I |
Inspection risk |
yes |
1 |
P |
Potential risk to patients |
Injury |
5 |
N |
Subsequent processes |
no |
10 |
S |
System category |
c - Configurable software package |
5 |
M |
Degree of maturity of the technology |
Internal company experience |
1 |
O |
Openness, Internet |
no |
1 |
RI = 1 * (5+3+5+3+3+0+1+5+10+5+1+1) = 42 |
The RI can now be used to determine the validation priority:
Low |
RI < 25 |
Medium |
25 Ј RI < 55 |
High |
55 Ј RI < 99 |
Equipment with the RI "0" result when no GxP requirements are placed on a system. Systems with no regulatory requirements are often found in administrative areas, e.g. a program for the administration of company parking spaces.
It is a good idea to perform a reality check using the empirical formula before it is globally implemented. Ultimately, the results of the RI are only a guide. Particularly in borderline cases, equipment or a computerised system can be assigned a different validation priority to that which would be determined by the analysis.
9.D.2.2 Measures that depend on the risk index
At every stage, the risk index enables efforts to be concentrated on the decisive points in the project. The following five tables contain the relevant guidelines depending on the system class and risk. In contrast to the risk index, the risk is only specified as low, medium or high.
Firmware
Deliverable |
Risk |
Level of detail / scalability of content |
---|---|---|
User SOP |
Low |
Manufacturer's instructions |
Medium |
Manufacturer' instructions: SOP, if use is complicated or error-prone |
|
High |
||
Change management |
Independent of risk |
SOPs for the different points are necessary and are not scalable in accordance with risk. This may be a general SOP that covers a wide range of equipment. |
User training |
||
Periodic review |
Standard systems
Deliverable |
Risk |
Level of detail / scalability of content |
---|---|---|
User SOP |
Low |
Manufacturer' instructions: SOP, if use is complicated or error-prone |
Medium |
System-specific SOP |
|
High |
||
Change management |
Independent of risk |
SOPs for the different points are necessary and are not scalable in accordance with risk. This may be a general SOP that covers a wide range of equipment. |
SOP for administration of physical and logical security |
||
SOP for backup and restore |
||
SOP for archiving |
||
User training |
||
Service level agreement |
Low |
System can be included with various other devices in a global service level agreement |
Medium |
||
High |
Service level agreement should include specific information on rapid reactions within hours. |
|
SOP for periodic review |
Low Medium |
SOPs for periodic testing are required This may be a general SOP that covers a wide range of equipment. |
High |
High-risk systems should be assessed more frequently. |
|
Retirement plan and report |
Low |
None |
Medium |
Simple plan |
|
High |
For a certain time period it should be possible to access the legacy system or reactivate it in an emergency. |
Configurable standard systems
Deliverable |
Risk |
Level of detail / scalability of content |
---|---|---|
Implementation plan |
Low |
Checklist with installation parameters for the individual workplace |
Medium |
System-specific implementation plan |
|
High |
||
User SOP |
Low |
Manufacturer's instructions |
Medium |
System-specific SOP |
|
High |
||
Change management |
Independent of risk |
SOPs for the different points are necessary and are not scalable in accordance with risk. This may be a general SOP that covers a wide range of equipment. |
SOP for administration of physical and logical security |
||
SOP for backup and restore |
||
SOP for archiving |
||
SOP for user training |
||
Service level agreement |
Low |
System can be included with various other devices in a global service level agreement |
Medium |
||
High |
Service level agreement should include specific information on rapid reactions within hours. |
|
SOP for periodic review |
Low Medium |
SOPs for periodic testing are required. This may be a general SOP that covers a wide range of equipment. |
High |
High-risk systems should be assessed more frequently. |
|
Retirement plan and report |
Low |
None |
Medium |
Simple plan |
|
High |
For a certain time period it should be possible to access the legacy system or reactivate it in an emergency. |
Standard systems with custom-developed supplements
Deliverable |
Risk |
Level of detail / scalability of content |
---|---|---|
Implementation plan |
Low |
Checklist with workplace settings |
Medium |
System-specific implementation plan |
|
High |
||
User SOP |
Low |
Manufacturer's instructions |
Medium |
System-specific SOP |
|
High |
||
Change management |
Independent of risk |
SOPs for the different points are necessary and are not scalable in accordance with risk. This may be a general SOP that covers a wide range of equipment. |
SOP for administration of physical and logical security |
||
SOP for backup and restore |
||
SOP for archiving |
||
SOP for user training |
||
Service level agreement |
Low |
System can be included with various other devices in a global service level agreement |
Medium |
||
High |
Service level agreement should include specific information on rapid reactions within hours. |
|
SOP for periodic review |
Low Medium |
SOPs for periodic testing are required. This may be a general SOP that covers a wide range of equipment. |
High |
High-risk systems should be assessed more frequently. |
|
Shutdown plan and report |
Low |
None |
Medium |
Simple plan |
|
High |
For a certain time period it should be possible to access the legacy system or reactivate it in an emergency. |
Custom-developed system
In terms of scalability and level of detail, the requirements for a custom-developed system are the same as the requirements for standard systems with custom-developed supplements. The difference is that these requirements apply to the whole system and not only to the supplements.
9.D.3 Risk management at the level of user requirements
For systems with a high risk, it is recommended to continue the risk analysis and also include the user requirements in the risk management process. Risk minimisation is particularly important in this context.
Table 4.2 in Annex M3 of the GAMP‚ Guide presents the following examples of strategies for minimising risk:
1. Changes to process or system design elements for reducing risk:
- Changes to the process design: One or more independent controls are integrated in the computerised process (e.g. additional data verification checks, plausibility checks, checksums as the last character when checking numeric entries).
- Introduction of all measures: Process changes external to the system may also make a contribution. "Double check" is the most widespread method.
- Changes to the product or system design are mainly made together with infrastructure changes. These include redundancies, error-tolerant work processes and uninterruptible power supplies, etc.
2. Changes to the project strategy for minimising risk:
- Persons who work on the project must have the appropriate training and qualifications in order to detect risks. This can be supported by the development of specific training and coaching program.
- The integration of testable deliverables and milestones can be used more frequently in higher-risk projects.
3. Risk elimination
- If the risk is particularly high for certain user requirements, in some circumstances these should not be implemented at all in order to completely exclude the risk.
Summary:
The simple classification of the GAMP‚ Guide only considers system complexity for the necessary validation activities, and differentiates between the five classes "operating systems, firmware, standard systems, configurable systems, and custom-developed systems". Certain deliverables such as the implementation plan, user SOPs, service level agreements, SOP for periodic review, and shutdown documentation, can be scaled according to the risk, in order to focus the validation activities on the points that represent the greatest risk.
Schьtz M. (Ed.): Risiko und Wagnis. Die Herausforderung der industriellen Welt. Bd. 1 und 2, Pfullingen 1990.
ISO/IEC Guide 73:2002 - Risk Management - Vocabulary - Guidelines for use in Standards.
ISO 14971:2000 - Application of Risk Management to Medical Devices.