Test Axioms - second attempt

This post is superceded by the book:

The Tester's Pocketbook

The latest (and all future) definitions of the axioms will appear here:

Test Axioms Website.

Here's my second attempt at a set of testing axioms. I've tried to name and define each with a 'one-liner'. The "Narrative/Action" makes a suggestion, and the "If you don't" suggests what might happen if you ignore the axiom. There are no recommendation as to the practice, process or heuristic you might want to deploy. Most axioms simply suggest you identify/agree/define an approach or appreciate/recognise a situation.

Axiom Name

Axiom

Action/Narrative

If you don’t

Stakeholder

Testing needs stakeholders

Identify and engage the people or organisations who will use and benefit from the test evidence you are to provide.

You won’t have a mandate or any authority for testing. You can’t report passes, fails or enquiries.

Test Basis

Test needs a source of knowledge to select things to test

Identify and agree the goals, requirements, heuristics, risks, hearsay needed to identify targets of testing.

If you don’t have a source of knowledge, how can you possibly select things to test? Randomly?

Test Oracle

Test needs a source of knowledge to evaluate actual behaviour

Define the sources of knowledge whether documented, physical, experience or hearsay-based to be used to determine expected behaviour.

If you don’t have a source of knowledge, how can you assess whether tested software behaves correctly or not?

Fallibility

Your sources of knowledge are fallible and incomplete

Test bases, oracles, requirements, goals are fallible because the people who write them are human. Your stakeholders may be responsible for providing imperfect or incomplete information because they are fallible too.

It is naive to think otherwise, as human error has an impact at every stage of the development lifecycle.

Scope Management

If you don’t manage scope, you may never meet stakeholder expectations

You must have a mechanism for identifying and agreeing the items in and out of scope (documentation, software or other deliverable or output) and managing change.

If you don’t manage scope, it is possible, probably even that stakeholders will assume you will test ‘everything’.

Coverage

Testing requires a coverage model or models

You must have a means of evaluating narratively, qualitatively or quantitatively the testing you plan to do or have done.

You may not be able to questions like, “what has been tested?”, “what has not been tested?”, “have you finished testing?”

Delivery

The usefulness of the intelligence produced by test determines the value of testing

Define a mechanism, frequency, media and format for the evidence to be provided.

Different stakeholders require different formats and analyses of intelligence so will not find inappropriate test reporting useful for decision making.

Environment

Test execution requires a known, controlled environment

Establish the need and requirements for an environment to be used for testing, including a mechanism for managing changes to that environment.

Environments may be delivered late or not at all or not be as required. This will delay testing or undermine the credibility of testing performed.

Event

Testing never goes as planned

Define a mechanism for managing and communicating events planned or unplanned that have a bearing on the successful delivery of test evidence.

Unplanned events can stop testing or adversely affect your plans and cause delay to testing, bug fixing or undermine your tests.

Prioritisation

The most important tests are those that uncover the best intelligence, fast

Agree a means of prioritising the tests to determine which of the infinite set of tests that could be prepared and run are prepared and run.

Stakeholders may not get the intelligence they require to make decisions because there is not time to run those tests.

Execution Sequencing

Run your most important tests first – you may not have time to run them later

Agree a means of sequencing the tests to ensure the ‘most important tests’ are run if execution time is limited or testing is stopped.

As/when testing stops, you may find that some important tests have not been run while less important tests have been run.

Design

Test design is based on models

Identify, adopt and agree a model or models to be used to select test cases.

Test design will be subjective, random and patchy.

Repeat-Test

Repeated tests are inevitable

Document and agree a policy for re-testing and regression testing

There may be no time in your plan for demonstrating fixes are done correctly and do not cause side-effects.

Good-Enough

Acceptance is always a compromise

Appreciate that the acceptance decision will always be made on partial information.

You may be frustrated because the system is imperfect; the testing is incomplete.

Never-Finished

Testing never finishes; it stops

Recognise that testing is time limited; testing may not complete; some bugs may not be fixed. Be prepared to report partial test results.

You may be unable to provide stakeholders with the intelligence they need because you are working to an unworkable plan.

The Sixteenth Axiom

Suggestions?

 

 

Hi Paul,

Good initiative! Here is some constructive feedback on the 3rd axiom (Test Oracle) of which I had some thoughts recently:

Doesn't this depend on the goal of your test activities. Most testing is indeed done to judge if certain behaviour is good or bad. But what if the goal is just to measure (e.g. performance). Not to see if it at least can handle a certain processing rate or does not exceed a pre-defined memory usage, but more to see what the value currently is. Some people (e.g. system architects) may be interested in these numbers, just to see how much room for development there is for next releases.

Paul Replies...
Yes, you have a point. It's common for non-functional requirements to be a bit 'thin on the ground' and in fact maybe half the time performance testing isn't requirements driven, as you say.

Let me think this through...

If there are no requirements or performance test objectives at all, the first stage is for the tester to create at least some load profiles. So work begins on getting some scenarios, transaction rates, database volumes etc. together. Let's call these load profiles. These need to be agreed by ones stakeholders as a reasonable basis for load simulations.

Then one needs to agree how one might vary loads around the 'design' average or peaks... Typically one creates a range of loads building up to design loads and beyond to provide a set of response time measurements for selected transactions against varying load. These can be presented as load-response time graphs.

Now, what is the required response time? Well, if they aren't defined ahead of time, I would propose some maximum acceptable response times. Finding this time on the graphs, we can then suggest maximum acceptable loads.

So now, we can discuss load profiles and anticipated response times with stakeholders. They might be lucky and it looks great - or unlucky. But at least now there's some hard data available for discussion.

Is there an 'oracle' - I think I would say that the forum convened to discuss the results are 'the oracle'. Ive worded the oracle axiom as, "Define the sources of knowledge whether documented, physical, experience or hearsay-based to be used to determine expected behaviour." With performance testing maybe it should read "... to be used to compare actual behaviour against"?

So, the oracle isn't necessarily used to predict the behaviour but could be if you have one in time ;-).

Now, as you say, architects might want to see how their beloved infrastructure performs. Presumably, they have some performance objectives to test against - otherwise what parameters did they use to create the architecture in the first place? (Did they just guess?) Anyway, if they want to simply find out what their architecture is capable of, I would call that a benchmarking or profiling exercise - so I would finesse that situation by saying, it's not really a test :-)

What do you think?


Hello Paul,
It feels like I volunteer on your journey to define Test Axioms. And I'm curious were it will end.

To understand your Axioms I tried to figure out some kind of structure like an organizational structure or test method structure. Hoping this helped me to identify some other Axioms.

I think all kinds of models are in these Axioms defined. Perhaps that is mandatory to create a list like this to prevent this list is guided by one model and input from others is forgotten.

Related to you 15th Axiom: You have also the opposite: Axiom Name: Never Exists Axiom: Testing never exists; it starts. Action: Acknowledge the moment you start with testing. This is the moment you have to question about other Axioms. If you don’t: starts testing you just do something. And what is not started cannot be stopped.

Somehow I’m missing a relation towards the team. As if no team is there no testing can be performed. If no skilled team is there axiom-names like: coverage and good-enough are answered differently.

I hope this give you some further input for your attempt.

With regards,
Jeroen

Paul replies...
The axioms aren't settled yet of course - but I do believe they are 'above' pre-defined approaches. That is, they are worth considering very early to help you understand the best way to approach testing. If the axioms are real, then they are true for any context - that's what I'm suggesting.

With regards you your suggested axiom... I'm not sure I understand. I think you are saying that the axioms need to be considered before you start (to even plan?) testing. I agree with that. I think this might be the value of the axioms as a set of challenges to your thinking about test. Maybe there's a way of wording it very precisely. I'll have to think about that one!

Comments

Actually, I had the benchmarking or profiling exercise in mind. The term I'm thinking of is measurment.

Since you call that not really a test, I searched for the definition of 'test' in the ISTQB definitions. It clearly states that a test is defined by, amongst others, the expected behaviour. I can perfectly live with the ISTQB definitions, since you need to have some sort of baseline.

In my opinion a test activity is about measuring quality. This measurement often results in verdicts (e.g. PASS or FAIL) but can also result in numbers (e.g. calls/s or %cpu). A number can be translated into requirement met or not, but that would throw away the extra information that the number holds, like if it is barely met or with lots of room for extra features. But I see now that this Axiom does not prevent to provide the stakeholders with more information than just if the requirements were met or not.

Paul Gerrard's picture

I'm worried you refer to the ISTQB definitions ;-) - the problem I have with these is they kind of come from nowhere and I must say the definitions don't really align with any English/US text book. I suspect the Dutch guys have over-influenced the ISTQB committee.

I don't doubt they speak better English than the British folk, but many of their terms don't appear anywhere except translated Dutch books.

Agreed about baseline. But the baseline does not need to be a written one. and yes, they can, in theory, be non-functional objectives.

The problem I have with qualiy, is, like TEST, there is no agreed definition. The most acessible definition of quality is something like, "absence of defects". Yuk!

This troubles me of course.

How can you measure something by the 'absence of something else?' Counting serious defects is fine. But how can you measure an attribute of something using a metric that involves counting things that no longer exist? I have a problem with that.

Anyway...

I still think a test is something that has a baseline, a predicted result etc. This may or may not be something documented.

With performance testing, one often has to execute an 'experiment' say, and then interpret the results. Not quite a test, I think.

Principal, Gerrard Consulting

Hi Paul

Great progress on this. I was thinking about some of the other "self evident truths" about testing and wandered if the following should be incorporated in some way:

- The "fundamental test cycle" is always followed.
In other word, no matter what type of testing approach/methodology a person uses, they will always plan, define, execute and report on their testing activities.

- Record what you do.
It is inevitable that you will need to record in some fashion what you you plan to do and then what you have done and and any defect found. To what level you record this depends on the nature of your business, but it is always there.

- Communicate your quantitative and qualitative findings
At some point, the tester will need to communicate what they have tested and found in an appropriate manner, to facilitate others to make a decision on what to do with the application under test. This would include some measures of testing and the testers gut feel.

Regards

Stevan Zivanovic

Hello Paul!

I tried to identify and understand the test axioms and yet, I can't fully construct the thoughts about what needs to happen in each axioms.

I must agree that most axioms simply suggest or define the situation. Yet, the problem in the situation is not clearly noticeable. Think one cannot assume for defining a goal or objective in the "axiom" column. My understanding is that the goal or objective are the primary concern(s) in making any change and/or suggestion of activities for axioms.

Think building the testing axioms requires a special form of preparation. And you need to know everything that an axioms is about, such as:

1. Identifying and understanding the situation - the axiom column. This includes the axiom goals, problems, and concerns (positive surprises).
2. Finding the solution to change forces or remove the blocks. Example: suggesting narrative and/or action.
3. Describing how the challenge is met.
4. Valueing the impact of the action.

Hope this could help.

Bong
Connection Practice - http://www.daniloalsonado.com/blog
QAST Practitioner Group - http://www.daniloalsonado.com/qastforum