What is formative and summative usability testing? Are they mandatory steps to release a medical device? Can Proof-of-Concept demos or clinical trials be used to ensure safe medical device usability?
Our customers often ponder the above-mentioned questions. This article sheds light on these issues based on the standard IEC62366 – Application of usability engineering to medical devices. The article also discusses how medical device usability testing differs from traditional usability evaluations, service safaris, demos and other methods that are in the toolbox of every user experience professional or service designer.
Usability testing is evaluation of a product or a concept with help of real users
Typically, in a protocol-driven usability testing, a real user of a product or a corresponding person uses a testable prototype or product according to a test plan. Test plan usually simulates a real workflow. A test facilitator conducts the test and observers spot potential problems of use, or other findings. Test protocol, amount of test personnel and test subjects, and steering of the test assignments is dependent for example on the open research questions, phase of product development and product complexity.
Formative usability testing (or formative evaluation) occurs during product design and development phase. Summative usability testing occurs in the end of product development.
Formative evaluation means evaluation of a user interface during product design and development
In formative evaluation, the strengths, weaknesses, and unexpected use errors are evaluated during product design and development. In this way the product is made to match user needs. From project perspective, use error and business-related risks are reduced before product launch.
Formative evaluation can be conducted for example through user interviews, observations, co-design workouts, use analytics of previous product versions, or protocol-based usability tests (described in the previous chapter). Especially evaluations in the beginning of product development are usually conversational co-design sessions.
Specialty of medical device formative evaluation is that the findings of the evaluation need to be analyzed also in terms of use errors and risks. Especially critical findings should be fixed before product launch. These findings and fixes should be linked to risk management in product documentation, where applicable.
Formative usability testing is iterative
It’s profitable to perform formative evaluations iteratively throughout a project. For example, in software development, formative evaluations can be implemented as a part of sprint cycles or as a part of Definition of Done readiness. In this way, the evaluation is automatically part of the product development process.
Formative evaluations in the end of product development can be regarded as “pre-summative” tests, with a purpose to pilot arrangements of the summative test and find out product risks before the actual summative test.
Summative usability test is a safety of use confirmation that occurs after product development
The summative usability test takes place after product development with the production-ready version. The product, related instructions, packaging, and related consumables or modules are tested in a realistic environment. The test is mandatory and should have 15 actual end users per each distinguished user group. If the product use requires training, similar training must be provided during the test with sufficient training decay. Remember – the idea is to find the potential use problems in a real-life-like situation.
In the summative test the focus is on the hazard related scenarios and workflows, identified for example during the design phase. The question is whether there are new unacceptable product risks found or whether the existing risk mitigations are sufficient during the test use.
Few years back we had a clinician as a test participant who arrived to the usability test directly from a tough nightshift. The tested product was a departmental healthcare IT solution. He had a lot of error clicks, difficulties on reading the user interface text, and remembering different sections of the system. Eventually he fell asleep in the middle of the test! This was an excellent example of a realistic use context and use errors – which would have been very difficult to simulate in a focus group in an office!
What is the clinical trial relationship to summative usability testing?
The goal of a medical device clinical trial is to verify clinical efficacy and evaluate the risk-benefit balance. By default, the clinical trial does not focus on ensuring product safety through objective usability methods. Because the clinical trial is conducted with real patients, safety of use must be ensured before the trial. Without comprehensive formative and summative usability testing, this might be difficult to assure.
Clinical trial is not equivalent to summative usability testing
However, if the safety of use can be justified, it is possible to integrate summative usability testing into the clinical trial with certain additional arrangements. There are synergies with the arrangement, especially in the context of low-risk devices and new innovations, as the clinical trial involves end-users, the right environment, and patients.
What about product reviews during the proof-of-concept phase?
Let’s jump to the beginning of the process… After all, product evaluations, demos, and evaluations by clinicians take place at the beginning of product innovation while Product/Market fit is still being sought. How do these activities relate to official usability testing?
One of the product innovation booby traps is the application of regulatory needs late in the product development. This happens often especially for new medical device players and can have bad impact on project schedule.
In terms of usability, the Product/Market fit evaluations at the concept phase should be done in agile, but comprehensive manner. Aim is to ensure that in addition to ensuring the product demand, concept and related business model, also potential use errors and their impact to regulation strategy are evaluated and documented. It also makes sense to take the credit of these early risk evaluations in the product documentation.
Use risk analysis is good product design but also good for business: safety of use and usability are great selling points.
It’s also a good practice to build a product evaluation rhythm within the concept and innovation phase, which, in addition to meeting the needs of users and stakeholders, supports the creation of a financial roadmap and the assessment of product feasibility.
Clinipower assists in usability testing and user evaluations
Clinipower plans and performs evaluations and user surveys during the innovation and concept design phase, as well as formative and summative usability tests. We also help implement user-centered interaction culture and process into organizations.
To support formative usability evaluations, we also created a Medical device software UI checklist, which is based on common medical device use risks and errors we’ve encountered during the years in the field.
If you have any questions, comments, or want to book a free product review session – please let us know.
