2016 Test Management Summit

The 2016 Annual Test Management Summit took place on Tuesday 26th April 2016 at the conference centre at Balls's Brothers, Minster Pavement

Sponsored by Worksoft and Delphix



Registration, Tea/Coffee


Welcome, Introductions


Session A

Gary Hallam, Delphix, "The future of Test Data Management"

Session B

Mike Jarred and Paul Collis, FCA, "The Evolution of Test Governance and Assurance"

Session C

Edmund Pringle, David Parker, Metaswitch, "Developing Excellent Technical Testers"




Session D

Jonathan Wright, Hitachi Consulting, "#DesignOps: DON’T PANIC! - The Hitchhikers Guide to Digital Engineering"

Session E

Steve Watson, Reed Business Information, "Diversification – is this the future for Test Managers?"

Steves presentation is here

Session F

Kevin Harris, New Voice Media, "Achieving Zero Bugs on the Test Environment" Kevin is presenting a webinar through EuroSTAR here




Session G

Rod Armstrong, EasyJet, Asia Shahzad, Hotels.com, "Brave New World - Workshop"

Session H

Rob Vanstone XebiaLabs, "DevOps Overview - Lessons learned for 2016..."

Session I

Graham Thomas, Independent Consultant, "Instructional Games for Testers, Test Leads & Test Managers"

Graham's slides are here




Session J

Mike Bartley, "Crowd Testing"

Session K

Isabel Evans, Independent, "Learning How to Tell Our Testing Stories"

Isabels' Slides are here

Session L

Dean Robertson, Worksoft, "Make Regression a Thing of the Past"


Pause - Closing Talk - "Tales of Assurance", Special Guest, Susan Windsor, WMHL Consulting


Drinks Reception






Summit Session Abstracts

Kevin Harris, New Voice Media, "Achieving Zero Bugs on the Test Environment"

We all know that we should be looking to test earlier at every opportunity, and that bugs that are found later in the development process cost more to fix.  This is a 75 minute Forum discussion to talk about how we could now be at the point where we should be able to prevent bugs from even getting to the Test Environment.  Is it possible to have bug-free quality code on the Test Environment?  If not, how far away are we, and what improvements can we look at to enable us to get there.

Possible steps to make this happen include: better communication about scope of testing at Kick-Off; striving for smaller stories; comprehensive test automation; Development & Test pairing; using Specification by Example (or alternatives); testing on Development machines (including remotely); reviewing and feeding back on automated tests in Development; running 5-Whys on all bugs found in Test; teaching your Developers how to test.

Do you feel your Test Team could get to this point? If so - then how soon? If not – then what’s blocking you?


Rod Armstrong, EasyJet, Asia Shahzad, Hotels.com, "Brave New World - Workshop"

In our interactive session Brave New World we will introduce a thinking tool that will allow you and your organisation to not only accept change, but to actively embrace it and follow a flow that will take that change through to a conclusion. We will illustrate the concepts with some examples of successes and also with some of significant failures within organisations that did not successfully apply this kind of model.  

We anticipate that this tool will be relevant to everyone no matter what their specific level or role is within an organisation and when applied can lead to you looking at all situations, both professionally and personally with a new perspective.


Edmund Pringle, David Parker, Metaswitch, "Developing Excellent Technical Testers"

As a company we place a huge emphasis on helping people develop to their full potential.  We have a great track record of helping new starters develop into solid testers and also do well developing testers on to become test managers or moving on to other roles in Development or Support.  However we have found it more challenging to help good testers who don’t want to move into management roles to become exceptional technical testers.  Part of the issue is that we didn’t have a great articulation of what a really great technical tester looked like to help inspire people.

To combat this, we put together a roadmap of testing skills that help people see where they are now and what they can push towards.  We’ve found this useful but it’s far from perfect.  We’d like to open it up for discussion, to share our ideas with a wider audience and to get some feedback to help improve it.


Jonathan Wright, Hitachi Consulting, "#DesignOps: DON’T PANIC! - The Hitchhikers Guide to Digital Engineering"

Join Jonathon Wright, Director of Digital Engineering at Hitachi to explore the digital delivery approaches including DesignOps that support modern day scaled agile and lean practices. It is estimated that 50 billion devices will be connected by 2020, providing data and analytics to direct new business models, personalised experiences and social innovation. So ... Don’t panic! Jonathon will share recent industry case studies testing Enterprise of Things (EoT) platforms including Smart Cities, Transportation and Energy.

Key Takeaways

  • Digital Disruption - accelerate the response to digital marketplace challenges
  • Digital Evolution - continuous assessment, insight, deployment and delivery 
  • AdaptiveUX - how to move between bi-modal (coreIT and fluidIT) to tri-modal digital delivery 
  • DesignOps - design practices (fail-fast experiments) through to operational feedback (measure/monitor)
  • DevOps - organization-wide cultural adoption of DevOps behaviours, processes and technology
  • Disciplined Agile - optimize the solution delivery life-cycle and deliver operational improvement


Graham Thomas, Independent Consultant, "Instructional Games for Testers, Test Leads & Test Managers"

The educational world has now recognised that we learn best when we are relaxed and having fun, where we have the freedom to experiment and learn through making mistakes.

This session introduces the audience to some instructional games that will help them, as testers, test leads and test managers, better understand the power of communication, and improve their communication skills.

Ten Hundred Words

The first game, the ‘Ten Hundred Words’ game, based on the #upgoerfive https://xkcd.com/1133/  is where we will describe testing, and test concepts, by only using words on the Ten Hundred most common words list – by the way thousand is not on this list. This helps us to communicate effectively, whilst not confusing our audience with technical double-speak (or jargon).

Test Concept Charades

The second game, ‘Test Concept Charades’, is where we have to explain common testing concepts without using the key words. Sounds simple until you try it and you have to explain complex concepts in plain language.

Both games combine to help us learn new and enhanced ways to communicate more effectively with others, especially with those who may not have the same level of technical understanding that we do.


Isabel Evans, Independent, "Learning How to Tell Our Testing Stories"

When we interact with other people we tell stories – with our words, our pictures, and with our faces and our bodies… As Terry Pratchett noted, we would better to call ourselves Pan narrans ‘the story telling ape’ rather than Homo sapiens ‘the wise man’

As IT practitioners on projects we can find that our messages are not heard or are misinterpreted. Sometimes we do not hear the messages other people need to tell us. Our natural abilities at storytelling and at appreciating the stories others tell us can be crushed under the weight of the ways we are expected to plan, report and communicate about the testing in IT projects.

In this workshop, I want to examine our ability to tell stories and how our natural delight in narrative can help us communicate about testing and quality to others.

We’ll look at why we tell stories, how we tell stories, and telling our stories in a way that is appealing to our audience. That means thinking about the role of oral, written and pictorial representations of stories, using the analogies of novels, serials, short stories, picture books, poems and songs. You’ll start to think about what story formats work best for your testing messages and how to adapt your testing stories to your audience.

We’ll also examine how we can better listen to other people’s stories about their parts of the project, about quality and about testing, and how we need to adapt our listening style to different storytellers when we are the audience.

I will tell stories and you will tell stories: we’ll listen to each other and learn to improve our listening and our narration.


Dean Robertson, Worksoft, "Make Regression a Thing of the Past"

Business and IT teams face intense pressure to deliver rapid change across complex enterprise landscapes. As a result, regression testing has become incredibly difficult to manage using manual and automated script-based approaches to QA. Fortunately, there’s a much better way to ensure every critical business process works as planned – even when applications change. Learn how high-velocity testing can help ensure unmatched quality through daily “lights-out” testing, shorten project timelines by more than 40% and eliminate thousands of hours of manual labor cutting test cycles by 90%+.


Mike Jarred and Paul Collis, FCA, "The Evolution of Test Governance and Assurance"

In 2015 Test Governance required a review as there were perceptions we weren’t as effective at testing as we needed to be, and the adequacy of our controls to ensure relevant testing was questioned.

Perceptions regarding effective testing were addressed, (data won over opinion), but the Governance question persisted..

  • Programme risks affecting testing outcomes were not adequately managed
  • Existing governance model provided little confidence relevant testing was happening, or that delivery outcomes would be successful
  • What did agile mean for Governance, how we interfaced with scrum teams? Was there a changing risk profile we needed to respond to?
  • Could governance improve collaboration with suppliers?
  • I wanted expertise in Test Group to support our test managers in delivery, could Governance evolve into a valued mechanism for our stakeholders & testers

Our model is evolving, discussing this experience and questions below will inform this evolution and share thinking in the audience. Cake will be provided to celebrate the 50th TMF.

  • Without Governance, how do you know if you need it?
  • What are other peoples experience operating Governance?
  • What are their TOR, how to measure success?
  • What are the inputs to Governance?
  • How important is cake, the benefit of serving it in Governance meetings?
  • With test automation becoming pervasive, how is the quality of test automation assured against understood risks?
  • The role of governance in DevOps?


Mike Bartley, "Crowd Testing"

Start-up success is about gaining traction with new ideas, which iterate quickly and persistently. Testing in this environment is challenging. To bring the right level of quality to the curious new customers, while ensuring new features and ideas make it to market quickly.

In this Customer led knowledge share we demonstrate how a crowd-based approach has been effective in balancing the challenges, and providing a long-term, scalable solution for the start-up as it grows.

The session will also discuss how Crowd Testing can add value to the QA performed by larger organisation


Rob Vanstone XebiaLabs, "DevOps Overview - Lessons learned for 2016..."

As DevOps and continuous delivery slip further into the mainstream, the question for most enterprises is becoming less “What is DevOps” and more “How do we get started?”

Hear from Rob Vanstone from XebiaLabs, as he looks back at lessons learned from implementing DevOps and Continuous Delivery initiatives in 2015 and what it means for 2016. Discover key takeaways that can help you make 2016 the year of DevOps in your organization.

The discussion will focus on

  • Skills needed for DevOps at Enterprise Scale
  • Balancing culture and tooling
  • Actionable advice from “Real World” implementations
  • Avoid pitfalls that delay your transformation


Steve Watson, Reed Business Infoomation, "Diversification – is this the future for Test Managers?"

I have been involved in testing for 28 years, and  a Test Manager for the last 8 of them. Testing is ingrained in me – cut me open like a stick of rock and you’ll probably see the word ‘Tester’ within! So the talk over the past few years about the future of Test Managers in an Agile world has been both interesting and alarming. Have I worked my way up to a role that will no longer exist in 5 years?

Who knows, but I was given the opportunity to try something different alongside my TM role last year, and I grabbed the opportunity with both hands, even though it took me outside of my comfort zone. I want to share my experience of becoming responsible for delivering a brand new product to market, despite having no track record in this area, the lessons I have learned, new skills developed, old skills sharpened, and mistakes made.

We will have an open discussion about the skills we as Test Managers possess that could help us diversify into other roles, and how it can benefit our careers.


Gary Hallam, Delphix, "The future of Test Data Management"

All companies experience frustration in getting regular up to date copies of good data into test environments. It can take days, weeks even months to provision and often all you are given are data subsets, a poor substitute. In the testing process it can take hours to reset the environment ready for re-testing and attempting to obtain data from archives for point-in-time analysis can be nigh impossible. Finally you know that the data should really be protected using masking, but this adds further weeks to the process (if done at all) and is massively expensive and complex.

What if all of this could be done using self-service in minutes? What if every tester could have their own complete and current copy of the data? What if they could refresh, reset, rewind or share that data whenever they wanted - without ever picking up the phone to operations. And what if data masking was just a seamless part of data delivery?