After seven years of working as an embedded systems tester for various devices in automotive and industrial automation areas, I thought about summarizing some of my experiences in this field. This isn’t by any chance a complete or exhaustive list of items, though it comes from many different test campaigns that I took part in or led. During this time, I also researched the subject vigorously trying to learn and implement good practices to the process. These experiences have equipped me with insights and lessons that have shaped my perspective on the art and science of testing. However, the field is so vast and continually evolving that I still continue to encounter surprising facets.
I would like to shed light on the seven key lessons I learned over these years. They represent an assortment of insights and nuggets of wisdom distilled from my hands-on experiences, supported by empirical data and case studies. Whether you are just embarking on your testing journey, or are a seasoned veteran in this field, I hope these lessons resonate with your experiences and provide a fresh perspective.
Testing is not just an end game
Many people instinctively think testing comes at the very end of the development cycle. It seems fair, considering you need the code or the product to be delivered to you before you can examine and verify it properly. But, in fact, testing doesn’t just happen at the end. It starts way before you even write your first test. There are many testing activities that should happen before you actually design your tests or begin to execute them.
The key to making things easier down the road is to get everyone on the same page about what the product is and what it’s supposed to do. It leads to eliminating a lot of time wasted later when people struggle to find that information that’s usually scattered around many different places. Another crucial aspect is defining expectations and deliverables for all engaged teams in the context of testing. Trust me, I’ve learned this the hard way!
The last part that I wish to learn way sooner during leading testing campaigns is that the more effort you put into static testing of requirements, the more it pays off later. There is one situation I’ll remember for years, where testers were not engaged enough in the requirements reviews which led to a few months of struggle when a few hundred of them had to be modified and then corresponding tests updated accordingly. Needless to say, this wasted a lot of time and could have been easily avoided.
Build trust and engage stakeholders
The dynamics of testing progress are influenced by an array of factors, such as shifting schedules, hardware failures, defect investigation times, and many more – I’m sure you could come up with some as well! This will all lead to delays and, if not communicated properly, corrode trust in the test team by people who are not that familiar with how things work on our end. I fell into this trap when taking over a project from a colleague and found out that we need much more effort than initially we thought. In this situation, I didn’t have a good communication plan that includes possible solutions to the issue already in place and the situation got escalated quite quickly.
Upon reflection, my communication strategy pivoted towards giving context, describing things in a clear and simple manner, and soliciting feedback. By consistently engaging stakeholders and keeping them abreast of potential risks and updates, the trust equation was significantly enhanced.
Know Your Product and Be Curious
I already wrote up a piece on that two weeks ago, however, let me briefly touch it here as well. You can’t use a one-size-fits-all approach when testing. Context-based testing is a testing approach where testing strategies, tactics, and plans are driven by the details of the specific situation or context. In my journey, I went from a fully abstract black-box approach to testing, to a high degree of context-based approach. The reason behind it is I realized how much efficiency increase it gives – it allows better prioritization of crucial tests to happen, focuses on what customers really want from the product, and gives more time to things that actually matter. That change came from researching the implementation of exploratory testing in my team’s testing process. I soon realized that it helps in finding very severe anomalies that could not be found by following a fully scripted testing approach. Knowing how your product is going to be used, and under what conditions, gives you great leverage in optimizing testing efforts, which is much needed in current highly dynamic development cycles.
An illustrative example of the efficacy of context-based testing can also be seen in the case of Microsoft’s Xbox team. The team uses telemetry data from users to understand real-world usage contexts and then formulates its testing strategies accordingly.
Testing and Development Go Hand in Hand
It is quite obvious that testing should be aligned with product development efforts, but I don’t think many people realize that testers can or even should influence the development priorities and cycle, so the most crucial features or ones that should be the most exhaustively tested, are developed sooner rather than later, if possible. Testing and development should not exist in silos but instead should be intertwined and aligned. I found it quite efficient when cross-functional teams are formed that consist of both testers and developers. This idea underpins the Agile development methodology followed by leading tech companies like Spotify and Amazon.
Touching on the topic of context once more – if we have it established, combined with product vision and its common understanding across the involved teams, we are able to come up with much better roadmaps for creating the product collectively. Developers will know what testers need and vice versa. I have found this approach highly effective when collaborating with development teams and having agreements in place for test framework inputs or scheduling defects verification and resolution cycles.
Automate Smartly, Not Just Blindly
Many people would disagree, but I don’t think it is worth automating every single test case. First of all, there’s a pesticide paradox – after running the same scripts over and over again you won’t find any new defects, same as pesticides stop affecting the parasites on the crops after a while, because they become resistant to it. But test automation is crucial when it comes down to exhaustive regression testing, large database testing, or all kind of activities that would require disproportionately much effort when executed manually.
I’ve been involved in testing some one-shot features or products where investing effort into test automation would not make any sense, as testing could be done much faster and more thoroughly manually. Test automation isn’t free. It comes with a price in the form of maintenance effort, which is easy to forget when excitement takes over. Notable companies like Google adhere to this selective automation approach. They prioritize automation based on potential time savings, thereby optimizing testing efforts.
Know When to Bend the Rules
The human aspect is very often brought up in the discussion about testing. There are plenty of interactions and communication between testers and other stakeholders, developers, and managers. What I find great and powerful about the testing process is that it can have a very positive impact on other efforts, enforcing certain behaviors or artifacts to be created. It can for example drive the requirements development and review process, improving things also for coders.
But what I also like to underline is, that behind every process, there are human-to-human interactions and it’s good to make an exception from time to time, if that helps the greater good. We can verify and resolve an anomaly quicker, or make an ad-hoc check for someone, work on lending a piece of equipment… There are many ways to move things forward this way and I strongly encourage you to be a person that always asks the game-changing question: “How can I help?”. It’s okay to bend the rules if keeping that in mind.
Keep an Eye on Your Progress
The fluid nature of testing makes it essential to continuously track progress and adjust strategies accordingly. Metrics such as burndown charts, defect counts, and test coverage provide actionable insights to navigate or preempt obstacles. When leading testing, I was especially focused on burndown charts, because they gave me information about any impediments almost immediately. Seeing a test blocker, I could work with the development team on resolving it or adapt and focus on different features that were not impacted by it.
It is also important to work closely with the team and listen to what they are struggling with. Collectively you are in a much better place to solve any issues that might occur.
Conclusions
Over these seven years, each experience, interaction, and challenge has imparted a lesson, shaping me as a tester and a leader. From understanding that testing starts before the execution phase, to knowing when to make exceptions to processes, to continually monitoring progress with metrics – these lessons have been instrumental in my journey.
However, it’s essential to remember that this list is not exhaustive. Testing is a dynamic field that continues to evolve with each technological advancement. Learning is continuous, and the thirst for knowledge should never cease. Whether you are a beginner or a seasoned professional in testing, I hope my experiences offer you a fresh perspective and valuable insights.
I invite you to share your thoughts, experiences, and lessons learned in the comments below or reach out to me directly. As we exchange knowledge and learn from each other, we continue to grow and evolve, not only as testers but as individuals, contributing to the progress of this intriguing field of testing and test management.