Embedded systems are the backbone of modern technologies, from Internet of Things (IoT) devices to critical infrastructure such as automotive systems and medical devices. As the demand for highly reliable and robust embedded software continues to grow, traditional testing methods alone may not be sufficient to uncover hidden defects that can impact system performance, safety, and security. That’s where exploratory testing comes into play, offering a dynamic and adaptable approach that can significantly enhance the quality of embedded software.
So, what exactly is exploratory testing, and how can it be applied to embedded systems? It is a common understanding that software testing is based on predesigned test cases, but testing itself is, in fact, an exploratory activity (see: https://www.satisfice.com/exploratory-testing). A unique characteristic of this process is engaging the most out of human knowledge and intelligence in an activity of simultaneous learning, creating a test design, and performing product verification. It goes beyond scripted test cases and encourages testers to be curious, creative, and adaptive. It shows to be very effective in finding defects—usually, those which could escape any formal technique, have a significant severity, but maybe lower customer occurrence, or are simply harder to reproduce. It is even hypothesized that exploratory testing is more efficient than test case-based testing in recognizing functional failures since the testers are able to use their personal knowledge in designing tests and recognizing failures on the fly even with low levels of testing experience.
Implementing Exploratory Testing
Though exploratory testing can be used as an addition to script testing, the software industry was quite ambiguous about it, both praising but also criticizing its applications. What makes it difficult to classify is that it does not belong to any specific testing technique and it will not be constrained by a certain testing characteristic or aspect in actual testing. It can be applied in any testing phase or different test implementation, and furthermore, it can be combined with corresponding test technologies. It is often opposed to scripted testing but in fact, exploratory testing can be of varying degrees of exploration from fully exploratory to fully scripted. Thus, the question is not whether or not to apply it, but rather when to apply which level to achieve the desired outcome.
The effectiveness of exploratory testing in uncovering hidden defects in embedded systems can be measured through various metrics. Some of the key metrics that can be used to evaluate the impact of exploratory testing include the number of defects identified, defect severity, and defect turnaround time. By comparing these metrics with those of scripted testing, you can assess the effectiveness of exploratory testing in terms of defect identification, the severity of defects found, and the time taken to address and fix the defects.
The benefits of embedded systems exploratory testing go beyond just defect detection. Here are some key advantages of incorporating exploratory testing into your embedded software development process:
- Early Defect Detection: Exploratory testing allows testers to actively explore the software, uncovering potential defects early in the development process. This enables prompt addressing of issues before they escalate into bigger problems, reducing the cost and effort associated with fixing defects in later stages of development or after deployment.
- Real-world Scenario Simulation: Exploratory testing allows testers to simulate real-world scenarios that may not be captured in scripted tests. This helps in identifying defects that may only occur in specific conditions or scenarios, such as extreme weather conditions, unexpected sensor failures, or uncommon user interactions.
- Improved Software Quality: By proactively exploring the software and gaining insights into its behavior, exploratory testing helps in improving overall software quality. It allows testers to understand the system’s performance, robustness, and usability, enabling them to provide valuable feedback to the development team for further improvements.
- Enhanced Collaboration: Exploratory testing encourages close collaboration between testers and the development team. Through continuous communication, feedback sharing, and iterative testing, exploratory testing fosters a collaborative approach, leading to a better understanding of the system’s behavior and more effective defect identification.
The Framework
To maximize the benefits of exploratory testing, I suggest structuring it as a part of your process. This ensures the sustainability, maintainability, and portability of the implemented solution and will help you make better decisions on when and where to introduce it. Below I present the framework that I use typically:
- Determine the exploratory testing ratio/scale.
- Define the desired coverage.
- Collect inputs.
- Identify test oracles
- Develop test charters:
- Describe the objectives.
- Provide an overall summary.
- Determine test targets.
- Set the timebox.
- State the charter vs. opportunity ratio.
- Execute scenarios (optionally – record the execution).
- Collect and analyze the results.
This approach involves several key steps to effectively implement this testing approach in embedded systems development. First, determine the exploratory testing ratio/scale, considering the optimal balance between scripted and exploratory testing. A more significant portion of scripted testing is usually a better fit for similar, repetitive projects, as it enables bigger-scale regression testing automation.
Next, define the desired coverage, identifying the areas or functionalities to be explored during testing. Consider familiarity with the device or feature under test, the possibility of introducing automation, existing documentation, or test engineers’ availability. Collecting inputs, such as system requirements, user scenarios, and real-world data, is required to inform the testing process. The advantage of exploratory testing is that it uses experience and expert judgment, therefore sometimes when you lack the proper level of documentation, you can use your own past experiences as input. Nevertheless, you have to identify test oracles, which serve as reference points for evaluating the correctness of the system behavior.
After those preparation steps, test charters development can be started. The test charter describes the objectives, provides an overall summary, determines test targets, set the timebox for testing, and states the charter vs. opportunity ratio. Finally, you can execute scenarios, record the execution if desired, and collect and analyze the results to identify defects and provide feedback for further improvements. Embedded systems exploratory testing can be executed systematically and effectively by following this framework to enhance software quality and system reliability.
Tips and Examples
To effectively incorporate exploratory testing into your embedded software development process, here are some best practices to consider:
- Train and Empower Testers: Provide training and resources to testers to develop their exploratory testing skills, such as critical thinking, problem-solving, and domain knowledge. Empower testers to actively explore the software, ask questions, and challenge assumptions during testing.
- Define Objectives and Scope: Clearly define the objectives and scope of exploratory testing for each testing session. Identify the specific areas or functionalities to be explored, the scenarios to be simulated, and the expected outcomes.
- Collaborate with the Development Team: Foster close collaboration between testers and the development team. Establish regular communication channels, share feedback, and engage in discussions to ensure a shared understanding of the system’s behavior and potential defects.
- Document and Share Findings: Document the findings from exploratory testing sessions, including defects identified, their severity, and the steps taken to reproduce them. Share these findings with the development team for further analysis and resolution.
- Continuously Improve: Learn from each exploratory testing session and continuously improve the testing process. Incorporate the insights gained into future testing sessions and refine the exploratory testing approach based on the feedback received.
Imagine you are testing a smart home security system, which consists of multiple interconnected IoT devices, including sensors, cameras, and a central hub. With scripted testing, you may cover the expected scenarios, such as arming and disarming the system, detecting intrusions, and sending alerts. However, exploratory testing can take your testing to the next level.
During exploratory testing, you may decide to simulate various real-world scenarios, such as attempting to hack into the system, triggering false alarms, or testing the system’s response to unexpected events, such as power outages or network disruptions. By actively exploring the system and its interactions, you may uncover vulnerabilities or unexpected behavior that may not have been captured in scripted tests.
For example, during exploratory testing, you may discover that the system’s camera fails to properly handle a specific lighting condition, leading to blurred images and inaccurate motion detection. This defect may not have been identified in scripted tests as it was not part of the expected scenarios. By identifying and addressing this defect early in the development process, you can improve the system’s performance and reliability, ensuring a higher level of security for end users.
Embedded systems play a crucial role also in the automotive industry, powering various functionalities such as infotainment systems, driver-assistance features, and safety-critical systems. As safety is of paramount importance in the automotive domain, thorough testing is essential to identify potential defects that may pose risks to passengers, pedestrians, and other road users.
Exploratory testing can be particularly effective in uncovering hidden defects in automotive systems. For instance, during exploratory testing of a vehicle’s infotainment system, you may decide to actively explore the system’s user interface, functionality, and interactions while driving. By doing so, you may identify issues such as unresponsive touchscreens, incorrect voice commands, or distracting visual displays that can impact the driver’s attention and safety.
Furthermore, exploratory testing can help in simulating real-world scenarios, such as extreme weather conditions, GPS signal loss, or unexpected sensor failures. By actively exploring the system’s behavior under such conditions, you may uncover potential defects, such as system crashes or incorrect sensor readings, that may not have been captured in scripted tests. Identifying and addressing these defects proactively can prevent safety hazards and ensure the overall reliability of the automotive system.
Conclusions
Embedded systems exploratory testing is a dynamic and flexible approach that can uncover hidden defects, provide valuable insights, and enhance overall software quality in embedded systems development. By actively exploring the software, simulating real-world scenarios, and fostering collaboration between testers and the development team, exploratory testing can significantly improve the effectiveness of embedded systems testing efforts.
Incorporating exploratory testing into your embedded software development process can lead to early defect detection, improved software quality, and enhanced collaboration among team members. Testers can proactively identify potential defects early in the development process, reducing the cost and effort associated with fixing issues in later stages of development or after deployment. The simulation of real-world scenarios during exploratory testing can uncover defects that may only occur in specific conditions, such as extreme weather conditions or uncommon user interactions, providing valuable feedback for further improvements.
Moreover, the collaborative approach fostered by exploratory testing encourages close collaboration between testers and the development team. Continuous communication, feedback sharing, and iterative testing enable a shared understanding of the system’s behavior and potential defects, leading to better software quality and more reliable embedded systems.
To effectively implement exploratory testing, it is essential to train and empower testers with the necessary skills and resources, clearly define the objectives and scope of testing, collaborate closely with the development team, document and share findings, and continuously improve the testing process based on insights gained.
In conclusion, incorporating embedded systems exploratory testing into your development process can significantly enhance the effectiveness of your testing efforts, leading to higher software quality, improved system reliability, and ultimately, a better user experience. Stay tuned for more insights and tips on embedded systems exploratory testing, and unlock the full potential of your embedded software testing efforts.