A comprehensive guide to ISO 14971: Risk management for medical devices
This is a guide for medical device professionals looking to gain a deeper understanding of the risk management process and how it may be applied. A fully implemented and well-designed risk management system can help your company not only comply with ISO 14971, but also improve outcomes throughout the product lifecycle.
Benjamin Franklin famously said “an ounce of prevention is worth a pound of cure” and it’s true. Identifying and mitigating issues early in the process is beneficial and may prevent costly design changes, production delays, and potential recalls later on. Following this logic, the same messaging that helps to build a culture of quality can help you to build a culture that embraces risk management.
ISO 14971 compliance quick checklist
Before diving into ISO 14971, do a quick evaluation of your organization. This will help you review if your quality management system (QMS) is prepared to support risk management activities in line with the standard.
- Do you have an established Risk Management Plan that covers the entire product lifecycle?
- Is your Risk Management File complete with risk analysis, risk evaluation, implementation of risk controls, and documentation of the risk evaluation and residual risk?
- Do you have evidence for the implementation and verification that risk controls are effective?
- Do you have evidence of a residual risk assessment?
- Do you have a system in place for routinely updating risk management documents post-production?
Guide to ISO 14971: Table of contents
- What is risk management?
- A history of ISO 14971
- Implementing ISO 14971
- Initiating risk management and design controls
- Part 1: Risk management plan
- Part 2: Risk Management File (RMF)
- Part 3: Risk analysis
- Part 4: Risk evaluation
- Part 5: Risk control
- Part 6: Evaluating the residual risk
- Part 7: Risk management compliance review
- Ongoing risk management
- Successful compliance with ISO 14971
- Final takeaways
What is risk management?
Risk management is a defined process for identifying potential hazards and mitigating them. When fully implemented, the risk management process starts simultaneously with the design process to minimize potential hazards from the beginning. The process continues through the product lifecycle and includes provisions for reviewing post-production data. The latter ensures that the device benefits continue to outweigh the risks and that any newly identified hazards are mitigated, if possible.
What risk management is not
- Risk management is not a process that can be managed by the Quality department single-handedly.
- It is not a process that can be completed once and then put on a shelf until a regulator asks to view it for a compliance check.
- It is not a process that can be tacked on at the end of the design process just to check the box.
A history of ISO 14971
ISO 14971 is the international standard that defines how to apply risk management to medical devices. The first edition of ISO 14971 was published in 2000. The current iteration is the third edition and was released in 2019. As the risk management standard has evolved it has increasingly focused on building risk into the quality management system (QMS) to fully integrate the risk process into the product life cycle. The most recent edition of ISO 14971 has added more requirements for the post-market risk management process, requiring manufacturers to analyze post-market data to better identify potential trends.
ISO 14971 is the definitive risk standard for the medical device industry and is the backbone to applying risk management in sub-processes such as device clinical trials. The rest of the regulatory landscape is finally catching up to the ISO 14971 revision, with a draft version of ICH Q9 Quality Risk Management released for public comment in December 2019.
What’s new in ISO 14971:2019?
One of the biggest differences in ISO 14971:2019 as opposed to previous editions is the increased requirements for post-market risk assessment. As part of a device risk management plan, manufacturers must establish a system to collect and analyze data about products once launched to market. Establishing these data collection channels is new to some manufacturers and may take creative thinking to solve for the best way to get device data.
ISO 14971 also specifically notes that the standard is intended to apply to software as a medical device (SaMD) and in vitro diagnostic devices (IVD). As SaMD becomes more prominent, these manufacturers need to understand risk management and how to apply it to their software devices.
ISO 14971: Key definitions
One factor that makes ISO 14971 somewhat confusing is the similar terms used throughout the standard with slightly different definitions. If you are struggling to keep your risk assessment straight from your risk evaluation this section is for you.
Risk analysis: This is the process of looking at a design or system and identifying possible hazards that could cause harm to people, property, or even the environment. Think of this as sitting around a table with your team and coming up with any possible, often seemingly silly, hazard that could arise.
Risk estimation: This is the process used to assign a numerical value to the laundry list of possibilities you came up with during the analysis. The estimation process considers the probability of occurrence and the severity of the harm.
Risk: Can be numerical or translated to a scale like low, medium, high, but this is the actual representation of the risk estimation for a given harm. So for example, a harm with a high severity but a very small probability of actually occurring may be a low risk. Sometimes this is referenced as a risk index.
Risk evaluation: This is a process of taking the risk analysis and comparing it to a set of predefined risk acceptance criteria. Usually this results in a risk evaluation summary document.
Risk assessment: This is a more comprehensive document that contains both the risk analysis and the risk evaluation.
Risk control: This is the step where the full risk assessment is reviewed and steps are taken to reduce risk to an acceptable level.
Risk management: The overall process of all things risk. Risk management is the umbrella term that refers to all the sub-processes and components.
Risk Management File: The records for all things risk management including analysis, estimation, acceptable risk ranges, mitigations, etc.
Implementing ISO 14971
All medical device manufacturers must have procedures for risk management. Existing procedures should have been reviewed and updated after the 2019 revision to update them to meet the requirements of the standard. Having procedures is great, but without buy-in and a company culture that supports risk management, your process will be unable to achieve full implementation.
Initiating risk management as part of early phase design controls
Too often manufacturers try to conduct all risk management activities late in the design controls process. Not only does this limit the ability of risk management to improve your design, but it’s not in compliance with the requirements of ISO 14971 or ISO 13485. Initiating risk management early in the design process is not optional.
The design controls process that is required as part of a QMS ensures that device development is carried out in a strict and organized manner. This process verifies that regulatory and user requirements are met throughout the development process.
By integrating risk management into these early design decisions, hazardous situations can be considered early on and may be mitigated through design choices if needed. This requirement is built into ISO 14971 but is also cross-integrated in ISO 13485 directly. ISO 13485 (section 7.3.3) specifies that outputs of risk management are one of the design and development inputs. This requirement forces manufacturers to conduct risk management processes during the design phases and use information gleaned from that process in the decisions for the design and development of the device.
Part 1: Risk management plan
Developing the risk management plan should be one of the first pieces of your design and development process. Each medical device must have its own risk management plan that identifies the risk management activities that need to occur at each phase of the product lifecycle. Further, the plan needs to establish how the device risks will be evaluated to determine if the risks are considered acceptable. The plan should be updated periodically, but starting with a solid initial plan can minimize hiccups down the road.
Part one of the plan is a guide for the who, what, where, and when of the risk management process for the specific device. In practice, this will likely be a table and will clarify which activities need to occur in each design phase as well as which activities will need to continue into the post-market phase. It’s important that the responsibility for each activity is clearly defined so that there is no ambiguity later on in the process.
The plan is a living document that should be periodically revised, but the initial plan should clearly lay out who needs to conduct each of the risk management activities and when it must be done. The timing is especially important for risk activities that build upon one another. For example, the risk estimation cannot be completed if the risk analysis has not yet been completed. Figure 1 shows a sample table identifying which activities need to be completed at each design phase and which department is responsible for ensuring that it is completed.
Figure 1: Risk Management Plan Table of Activities
|Risk management activities||Primary responsibility||Phase 1 - Planning||Phase 2 - Development||Phase 3 - Verification||Phase 4 - Validation||
|Risk management plan||QA||X||X||X||X||X|
Part two of the risk management plan describes how the process is going to be applied. It must include the following pieces:
- Review requirements for each risk management activity
- Who must conduct the review? What does the review consist of? Any approvals that are required.
- Risk acceptability criteria
- What levels of risk are acceptable? How will acceptability be determined if a quantitative probability cannot be assigned to the hazard?
- Note: These criteria must be defined BEFORE risk analysis and estimation to keep the process as objective as possible.
- Residual risk acceptance criteria
- What level of residual risk as acceptable? How will residual risk be identified and determined?
- Plan for verification of risk controls
- How will it be verified that risk controls have been implemented into the process? How will these controls be reviewed to determine effectiveness at reducing risk?
- Plan for collecting and reviewing post-production information
- What sources will be used to gather post-production data? How will that data be reviewed? How will it be used as an input to the ongoing risk management process?
Risk acceptability criteria
It’s important to define the acceptability criteria at the outset of the development process to make the risk analysis process as objective as possible. When established in the plan during the earliest design phase, criteria is less likely to be influenced by data that’s been acquired during the development process. Criteria can be quantitative thresholds based on a calculation of a risk index number. This risk index number should be calculated from the probability, severity, and any other metrics used to quantify the risk of the potential harm. The risk acceptability criteria will be applied before any mitigations or risk controls are put into place so a higher level of risk is generally acceptable at this point.
Residual risk acceptance criteria
The residual risk acceptance criteria is very similar to the risk acceptability criteria, except for when the criteria is applied. The residual risk acceptance criteria will only be applied after risk controls have been applied. This acceptability criteria will be applied only to any residual risk estimated after all mitigations have been completed. Failure to meet the residual risk acceptance thresholds doesn’t doom a device forever—it simply means that the risk must be further mitigated if possible.
Plan for verification of risk controls
A plan for verification of risk controls will identify how mitigations will be verified. The verification process should be similar to verifications that would be completed if a design change is implemented through the change control process.
The verification plan should consider if there are any validation activities that need to be completed to ensure that the mitigation did not cause unintended issues with another aspect of the process or design. The verification plan should also take steps to determine if the mitigation was effective in reducing the probability of the harm occurring. It may often be difficult to determine if mitigations were effective, but having a plan allows for a concerted effort to document and quantify the process where possible.
It’s worth noting that some mitigations will be already planned independent of the risk assessment process, but they can still count as a mitigation. For example, device labeling regulations require certain information and symbols to appear on the device labeling. This is a known regulation and expectation; however, warnings appearing on labeling can also be considered a mitigation for some potential harms.
Plan for collecting and reviewing post-production information
This is one area that manufacturers may struggle to comply with, primarily because it’s a newer requirement and there’s much more focus on it now in the current regulatory climate.
Traditional channels for receiving feedback for the risk management process are the process and product nonconformance system and customer complaints. The expectation is that this is taken much further and includes analysis of information from all levels of the supply chain, analysis of information about what is the current state of the art, and any publicly available information.
The post-production information collection plan should consider literature reviews, regulatory database searches for recalls, and possibly developing new channels through physician surveys or similar tools to gain feedback.
Part 2: Risk Management File (RMF)
Building your Risk Management File is much like building a Design History File or Technical File. The RMF contains all the evidence to show that you are identifying hazards, mitigating them, and further evaluating them once mitigations have been implemented. Specifically, the RMF must contain traceability for each hazard to the associated risk analysis, risk evaluation, risk controls, and evaluation of residual risks. This does not mean that the RMF must be a massive file full of every associated document, but it does mean that there at least needs to be clear references to the documents.
If labeling is used as a risk control measure, there needs to be a reference to the specific label, usually the document number. If there were any verification or validation reports generated, there need to be references to those document numbers as well.
Figure 2 is a sample excerpt of a simple RMF. It identifies the associated documents for the sample hazard. A document user could then look at each document referenced to verify that the risk was adequately controlled and that the residual risk was found to be acceptable or otherwise documented that the risk-benefit rationale determined the benefit to outweigh the risks.
Figure 2: Risk Management File Traceability Matrix
Implementation & verification
|Bacterial contamination||RA-0001, Risk assessment||
-Validation IFU LBL-23412
|RA-0002, Residual Risk Analysis|
Part 3: Risk analysis
Each medical device must have its own unique risk analysis. If there’s already a completed risk analysis for a similar device, this can be used as a starting point for the analysis but should not preclude further device risk analysis.
The analysis should be done by a cross-functional team, so that all perspectives can be considered. This may mean sitting around a table and coming up with ideas or working collaboratively on a shared document remotely. Your risk analysis must identify and describe the device being analyzed, who was involved in the analysis, and the scope of the analysis. For a new project, the scope will likely be very broad. Since risk analysis is completed throughout the product life cycle, later on the scope may be very narrow if you are analyzing a design change or process change.
When starting on a risk analysis for a brand-new project it may be helpful to start with a divide and conquer approach. You can divide the scope and the type of analysis up across the team. For example, personnel with greater knowledge of clinical use of the device might be best suited to the initial analysis for the intended use and foreseeable misuse. The quality/regulatory team may be well suited to look at safety since they are familiar with complaints, recalls, and the standards that apply.
Once an initial list of hazards is generated based on intended use, foreseeable misuse, and safety characteristics, the team should work collaboratively to identify hazardous situations. A thorough risk analysis requires creative thinking. At this stage it’s preferred to think of hazards and hazardous situations that seem highly unlikely rather than omit them. The more complete your risk analysis is, the easier it will be to mitigate risk and develop a safer, better device.
Sources for risk analysis
When trying to come up with hazards and hazardous situations it may be useful to consult publicly available information about similar devices already on the market. There may be information in their public complaint reporting data that will give some ideas for hazards that you may not have considered. This information will also be useful later in the risk estimation phase. IS0 14971 contains a good starter list of hazards in Annex C which is a solid starting point for developing a thorough analysis.
Hazards, hazardous situations, and harms
ISO 14971 defines a hazard as “a potential source of harm”. That definition is vague and—without context—isn’t useful. Only when you get further into the standard and the informative annexes is it clear how to differentiate between the terms and how they’re interrelated.
The hazard is a broader identification of something that causes some sort of harm. It’s worth noting that a hazard cannot cause harm unless there’s a trigger event or events, which is considered the hazardous situation. For example, bacterial contamination is considered a hazard. The existence of bacteria is not harmful to the device in and of itself, however, when that bacteria is not removed properly (the hazardous situation) before being introduced during a surgical procedure, the result could be a bacterial infection (the harm).
A single hazard may have many different hazardous situations and harms. When organizing your risk analysis documentation, it may be helpful to identify each hazardous situation with some sort of code for traceability documentation later. For example, all hazardous situations pertaining to bacterial contamination could be identified by an alpha code that identifies the hazard it represents and then a consecutive number to identify the line item. Hazardous situation number three on the list could be A.3, easily allowing document reviewers to know it is tied to hazard A.
Risk estimation can be identified in the risk assessment either qualitatively or quantitatively, or possibly a mix depending on the availability of quantitative data. A quantitative estimate that is a calculated probability of occurrence doesn’tt need further explanation in the documentation. If you are using a qualitative system such as high, medium, or low risk, you will need to define what those categories mean within your risk management documentation.
For each hazardous situation that is identified a risk estimation must be applied. Estimating the probability of occurrence of harm is not possible for all hazardous conditions. In cases like this you are able to simply identify the possible consequences. The risk estimation needs to consider both the probability of the harm occurring and the severity of the harm. You will typically then assign a risk index level to that hazardous situation. This could be calculated using an equation, or for a more qualitative system it could be assigned a general category of risk level.
To estimate risks, even qualitatively, there needs to be some sort of data or information to support that assessment. These sources may vary widely depending on the hazard under evaluation and how common a hazardous situation is in industry. Data from publicly available incident reports or published literature is easy to obtain and can be very specific, especially if there is already an existing device that is similar. If a device is brand new without similar competing devices it may require more reliance on reports from various pre-market testing and possibly even clinical trial or usability testing data. Relying on expert consultants to provide their opinion is also a valid source for estimating risk, however, be sure that the expert qualifications are documented within your quality system.
This is still the risk estimation stage, which doesn’t look at acceptability of the risk levels that have been identified, nor does it consider risk control measures that will be applied.
Part 4: Risk evaluation
To complete the risk evaluation it’s critical that the risk management plan has already clearly identified the acceptability criteria. If the criteria is clearly defined it should be a quick exercise in comparing the estimated risk level to the criteria to determine if it passes or fails.
If the risk fails to meet the acceptability criteria at this stage, there’s no need to panic—there can always be risk controls put into place. Some risks may be acceptable without any risk controls, but you still may want to apply risk controls to further mitigate the risk if possible. Think of it as extra insurance that the risk will not occur and also a means of improving the device.
The risk evaluation step will be revisited once risk controls are implemented and verified.
Part 5: Risk control
When a risk is identified, especially if the hazardous situation is likely to occur, risk control measures can be taken to reduce the risk of the hazardous situation occurring. The type of control applied can vary widely based on the hazardous situation and may include training, labeling, validation, and design characteristics to name a few.
ISO 14971 specifies that risk control measures should be applied in an order that puts the onus first on the design and manufacturing process through “inherently safe design and manufacture”.
This first layer will include things like selecting appropriate materials, ensuring devices don’t have sharp edges that could puncture a sterile barrier, or other characteristics designed into the product to minimize a foreseen hazardous situation. The next layer of control is features of the device or in the manufacturing process that are protective, such as safety guards, product markings, or quality control checks. The final layer of control is information for safety including labeling and instructions for use, and user training. This last layer of control is the least effective, because it relies on the user to do the right thing and the manufacturer does not have sufficient control over user actions to ensure consistency.
This preference for priority of controls is logical and requires manufacturers to build safety into the device rather than trying to compensate for an unsafe device later on with user training or workarounds. Again, starting with risk assessment at the beginning of the design process is critical for successful implementation of the risk management process.
Implementing risk controls
Once you have determined which risk control measures need to be implemented to reduce risk, they need to be not only implemented, but also verified. The process qualifications and validation that you’ll be doing as part of your standard design and development process can serve as the verification check for many of your mitigating factors. For mitigations that aren’t already being verified through another process, you will need to deliberately document that the measure has been implemented. Further, the verification of that implementation should include verification of the effectiveness of the control measure if possible. The evidence of the implementation and verification activities must be part of your Risk Management File and traceable to each hazardous situation identified during your risk analysis.
Part 6: Evaluating the residual risk
The final piece to fully addressing a hazardous situation is to review the situation after risk control measures have been implemented and verified. This evaluation process considers the risk that still remains for not only the individual hazards and hazardous situations, but for the overall device.
The residual risk will look at the expected benefits that the device confers on the user or patient when used according to its intended use. With the benefits in mind the residual risk is compared against the acceptability criteria to determine if the overall device risk is acceptable. Again, it’s important to note that this acceptability criteria should have been previously specified in the Risk Management Plan in order to maintain some objectivity within the risk evaluation process.
The overall residual risk may be documented by identifying the residual risk for each hazardous situation, then adding a statement about the overall risk-benefit rationale, and identifying if the risk was found to be acceptable. Any significant risks that still remain must be identified and disclosed. These residual risks will be identified in your Instructions for Use (IFU) for the device and will be added to your Investigator’s Brochure if the device is going through the clinical trial process.
So what happens if the overall residual risk is not meeting the acceptability criteria? If the residual risk is not acceptable you can go through another iteration of identifying, implementing, and verifying risk controls. The risk management process is cyclical, so you can always go back to a prior step to try to bring the risk levels down to an acceptable level. Alternatively, if the findings of your risk assessment are just not promising that you will be able to reduce the risk enough, the residual risk will be documented as unacceptable within the Risk Management File and the design and development halted.
Part 7: Risk management compliance review
The last step of the risk management process that occurs prior to device release is a comprehensive review of the entire process. This review should be built into your design controls system as one of the final checkboxes before pushing the device to clinical or commercial production. This process will most likely be assigned to the Quality or Regulatory team and will be much like an audit of the process.
This final risk review will verify that the risk management plan has been established and implemented, including the steps for risk management that will occur once the device goes into production. Part of that implementation should include reviewing your systems to make sure that there are procedures in place to make sure that the post-production risk management activities are taken care of. This review will also verify that the mechanisms for collecting and reviewing that post-production data have been implemented. This may mean establishing quarterly metric reporting for these criteria, annual device user surveys, or other tools for collecting data. These data collection channels need to be in place before the start button is pushed on the production process to ensure the data from the initial devices is captured.
Finally, the risk review will also verify that the residual risk evaluation is documented and that the findings concluded that the residual risk is acceptable. This will be confirmed by reviewing the documentation including the Risk Management File. This is just a secondary check that the paperwork is in order to certify that the device risk is acceptable as is.
How to conduct ongoing risk management
At this point, the channels for collecting post-production data should be established within your risk management plan. Since the risk management plan is a living document, it should be periodically updated as you identify new sources of information or determine that previously planned sources of information are not providing worthwhile data. If you need to look for new sources of data you should again consider the following sources:
- Information from the production process
- Think in-process nonconformance data, receiving inspection data, and data from any production tests that are routinely run.
- Information from the device user
- For electronic devices can you get user information from the device remotely? What about customer surveys included in the device packaging or conducted at conferences?
- Information from installation and maintenance personnel
- What issues are encountered during installation and maintenance? Are these your personnel maintaining sensitive electronic equipment or are they hospital personnel that are cleaning and sterilizing devices for reuse?
- Information generated by the supply chain
- Make sure that your distributors are passing information along from their customers (you may not find out boxes are being damaged in transit if this data channel is not activated).
- Publicly available scientific literature and regulatory reporting
- Device incident data can be a goldmine for your risk analysis processes. Scientific literature can include comparisons to a competitor device or user experience data.
- Information about the state of the art
- This type of information will come from conferences, technical journals, and industry publications.
- Are newer competing devices safer because of a design feature? Does your device risk acceptability need to be updated in consideration of the change in state of the art?
Collecting data and having established data collection channels is the most difficult part of the post-production risk management process. Once you have that data and information it can be fed back into the risk management process, starting back at the risk analysis.
You will need to establish a process for periodic review of the data and information along with a review of the existing risk analysis documents. The review process must also include escalation triggers for immediate evaluation of the risk analysis documents if concerning data or information is received. For example, if you have identified a higher-than-normal level of process nonconformance that could result in negative patient outcomes, that should trigger an immediate review of the risk analysis. This may also be in conjunction with evaluating the hazardous situation to determine if recall efforts are required.
For routine review of the post-production data, your risk analysis team will need to sift through the new information and data to determine if there are new hazards or hazardous situations that were not previously captured on your risk analysis. Further, they will need to determine if there is new information that suggests the documented risk estimate may not be accurate. For example, if the rate of a particular surgical complication higher with the device is higher than expected that estimate will need to be revised and the residual risk will need to be evaluated again.
When risk management input data is reviewed it must be documented and added to the risk management file. This documentation must include any decisions or actions that were taken and can be accomplished with meeting minutes, some sort of review form, or otherwise documented. The documentation will also include any revised copies of the risk assessment documents including the analysis, estimation, controls, and residual risk evaluation. The risk management plan may also need to be updated if changes to state of the art indicate that risk acceptability criteria should be changed or if other changes to the plan are needed.
The output from this review process should then also be fed into your management review process as an input. This information will be used by management to determine if the process is suitable and effective but does not necessarily need to contain all the details and minutiae of the risk assessment process.
Risk management cycle
The risk management process is cyclical and routine review activities will need to continue throughout the life cycle of the device. It is expected that the risk management burden will decrease as the device is on the market. New device data and information soon after product launch will result in more frequent updates to the risk assessment, with updates becoming minimal once the device has been on the market for several years and incident rates become more well established.
With each new device a new risk management plan and Risk Management File will need to be established. If an existing device is similar the risk management documents for that device can be leveraged as a starting point, but each device must have its own file with clear traceability. It may be preferable to stagger the routine review processes for each device so that the burden is more evenly distributed.
Successful compliance with ISO 14971
ISO 14971 has some big requirements and is seemingly a large burden on the quality and regulatory department. However, if the process is well implemented, it has the potential to reduce quality and regulatory burden through improved device design and safety resulting in less device nonconformances and device field incidents.
ISO 13485:2016 also requires risk to be considered throughout the quality management system, so when you are done implementing your ISO 14971, just keep going. Regulators continue to put increasing focus on ongoing risk management activities, so it’s time to embrace it and use the process to create a net benefit instead of just being another regulatory hurdle to jump. Focus on building a company culture that embraces risk management. Risk management can be used as a tool to prioritize ever-increasing workloads to assist with overall compliance.
If you’re looking for tools to help you with the risk analysis process, you may want to look at ICH Q9. Although it’s directed at drug manufacturers, it offers good tools such as Fault Tree Analysis to help with the actual work of risk analysis. Further, it offers suggestions for building risk management into your quality system for a truly integrated risk management process.
Final takeaways for risk management
- Build a strong foundation. Your risk management plan is the foundation of your entire risk management process for that device. By spending time and energy up front to develop a sound plan with adequate detail, you will save yourself from issues later.
- Don’t let risk management overwhelm you. Risk management can seem like a huge undertaking and a little confusing. Risk management can seem especially daunting if you are the person that is expected to manage the system and make sure everyone else understands the process. Make sure to do your research on the process so that you are confident that you understand the system. If you don’t have a good handle on it yourself, you won’t be able to explain it clearly to others.
- Remember that risk management is an ongoing activity. Doing the initial pre-market risk management process will get you to market, but you cannot stop there. Ongoing risk management processes must be built into your risk management plan, but even the best plan is just that unless someone makes sure it is adhered to. Make sure that you build a system that keeps regular risk review from falling to the back burner.
Need an eQMS built for ISO 14971 risk management? Take a look at Qualio.