The Feeling of Rejection

2 Comments

In scope or out of scope? This has got to be one of the most common issues that we face when raising defects. I raise an issue that then comes back as rejected because the documentation isn’t specific enough around that area and development is often focussed more on time and effort involved than usability.

There are four possible outcomes:

  1. The defect remains rejected and is ultimately closed
  2. The rejection is overturned, and the defect is accepted as valid, and fixed
  3. The rejection is valid, but leads to a change in the documentation, and in turn, a fix to the defect
  4. The rejection remains, but the issue is progressed to an ‘Enhancement’ or other suitable status, to be addressed as part of some future release.

Which outcome should you push for?

As with everything in our craft, it’s all down to the context.

Ask yourself five questions

  1. How likely is the defect to occur?
  2. How severe is it?
  3. How easy is it to fix?
  4. Is there a workaround?
  5. Is this internal, or external facing?

The combination of the first two questions gives you an ‘impact assessment’. You can go as far as scoring each answer and multiplying the two to give you a figure (management loves to report on figures rather than words).

The third question should then be considered in light of this assessment. If it’s easy to fix and has a high impact, then push for an immediate fix. If it’s very difficult to fix but has a low impact, you might want to let this one slide.

Question four adds a bit more qualification – if there’s no workaround available then you are in a better position to push for the fix now. If there is one available then maybe it can wait a while.

Question five is probably the most important one. If this is external, then it can be a real game-changer, as suddenly even the smallest of defects can cause real damage to a brand if exposed to the public. If it’s internal, then the opposite is true, large defects can then be handled by ‘business process’, ‘workflow’ or my favourite, as a ‘training issue’!

Consider a spelling mistake on a page. Normally it’s not a huge issue and a few of them often get through. Consider though that this is a website for a company selling English language educational assets. Having a spelling mistake on your front page is likely to cause people to think twice before purchasing from you.

Ultimately though, any software is a solution to a problem. If the problem isn’t solved, then the solution is broken. If the solution is broken, then a defect needs fixing.

Applied Epistemology

Leave a comment

Dipping back into ‘Lessons Learned in Software Testing’ after bit of a break from it, I rediscovered the topics on Applied Epistemology, namely that this is what testing is. It jumped out at me, because I’ve just encountered an example of it in my own testing.
As the book states, one topic within epistemology is ‘meaning and ambiguity in natural language’. Think about that for a minute or two, and I’m sure you’ll easily come up with examples that have affected your testing. My example is below:

The product I was testing holds address data, and as with any product that holds this kind of information, the data can easily become invalid or out of date, and need cleansing or maintenance. Part of this will include marking addresses as ‘Goneaways’. There are a number of ways of doing this, you could either compare your data to a screening file (in the UK we have a GAS, or Goneaway suppression file, for example) or you could rely on the fact that any post to a goneaway address should be returned. Since there’s no point sending out communications to these addresses, but given that legally we need to retain all information that we have for a set period, then it follows that we need a way to mark these addresses without deleting them. They’re not valid addresses for us to send to, and you might even go so far as to label them as ‘Bad’.

Unfortunately, that’s exactly what happened. Somewhere between gathering requirements, and producing code, ‘Goneaway’ got converted to ‘Bad’. Now, a label is just a marker, and if everyone had the same understanding, then this wouldn’t have been a huge problem. It became a problem, because ‘Bad’ was then interpreted to mean structurally incorrect and in need of correction. Of course, a Goneaway’s address is usually structurally sound, and is a valid address in the sense that the postal service can happily locate it, so to mark it as ‘Bad’ – as in incorrect – was not the correct solution to our problem.

‘So what?’ you might be asking at this point – well, the key is in the possible scenarios that would make use of this marker. Imagine that someone else moves into this ‘Goneaway’ address, but that this someone else is also on our database. If we associate this Bad Address to them, then we won’t send out any communications to them. If we unmark the address as ‘Bad’, then we’ll end up sending communications to both the new resident and the previous resident, at the same address.

Marking an address as a Goneaway subtly alters the requirement, as there is nothing wrong with the address itself – we actually want to flag the link between the address and the resident. That way, if the address is reused, then a new link can be created to a new resident, and this new link can remain unmarked. The use of the word ‘Bad’ led to ambiguity, and this ambiguity was successfully picked up by walking through test scenarios based on the original requirements.

Job ads

Leave a comment

A few days ago, I got the following email from a recruitment agent.

Hi,

I’m sorry if the email does not apply to you.

I’m currently working on behalf of a client who are looking for a Contract Systems Tester for an intial 3 month period to start immediately. The role is to help implement best testing practice across the company,please find a brief outline below

They need to be able to work on their own and not need handholding. Be able to supervise others in the team that are less experience. In brief their tasks will include:

– Writing of System Test Plans in compliance with the project Gantt and plan

– Writing of System Test Scripts, Conditions and Results

– Completion of test runs in accordance with Functional requirements and solution design

– Reporting of bugs and follow through to resolution

– Managing the customer during UAT reporting of bugs to technical staff and resolution thereafter

– Reporting of progress to the Project Manager

– Be deadline driven

Most of that is standard cookie cutter stuff, there’s certainly nothing that stands out and grabs my attention, nothing that would make me think, ‘Wow, I want that role’.

There were two lines though that I was struck by, but for the wrong reasons.

To help implement best testing practice

I’m more taken with the principles of the Context-Driven school of testing than ISEB/ISQTB, and they quite clearly point out that there is no such thing as best practices.

Maybe if you’re repeating the same task day after day, then there are some ways of doing things that will become ‘best’, but testing is about finding new things every day; looking at changes to existing applications as well as entirely new ones, always trying to come up with new ways of finding defects.

Be deadline driven

‘Deadline Aware’ maybe at a stretch, but deadlines aren’t what drive me, quality is. By that, I mean that I want to make sure that as many of the bugs in there have been found, and that the development team has had a chance to address them. I’d prefer to ship a higher quality application late, than a broken application on time.

(Note: There have been times when an application I’ve been testing has had to go live with serious known defects. This has usually been due to external influences, such as tax and month end issues, but this has so far proven the exception rather than the rule.)

I’m also compelled to point out the typo on ‘experience‘ – we testers do tend to have an eye for detail! (And yes, I know that’s not the only issue with the text.)

It’s a tough economic climate out there, with hundreds of applicants chasing every application. Does that really justify ads like the above?

Mr. Monty Hall

3 Comments

OK, so in my last blog I posed a question, and the answers that came back got it right – which I was hoping for, but wasn’t expecting. This leads me to believe that those brave enough to comment (as WordPress tells me that there were a lot more views than comments!) are more logically-thinking than average – or have possibly just heard of the problem before!

In doing a bit of research before writing this half of the post, I realised that this ‘hypothetical’ scenario I’d been taught was actually a real one, from a real quiz show aired in America. The host of that quiz show was named Monty Hall, and in his case, the prizes were goats and a car, rather than money and beans. The underlying puzzle goes even further back – full details can be found on Wikipedia.

The basic answer of course, is that you should take the opportunity to move, every time. You started out with a one in three chance, and Monty gives you the option to switch to a two in three chance.

According to Wikipedia, when this was first discussed thousands of people, many with PHDs, disagreed and insisted that it didn’t really matter whether you switched as the chances were now 50:50. I found the same when discussing this with colleagues. Some others insist that you’re better off staying where you are, although I’ve never really got a rational explanation as to why they thought this.

The reason why it isn’t 50:50 is because the door Monty chose wasn’t chosen randomly. He couldn’t choose the door you were already on, and he couldn’t choose the one with the prize behind it – as otherwise he can’t ask you if you want to switch. Because of that, Monty is limited to choosing the losing door, which fundamentally alters the statistics of the choice you’re then given.

The fact that many people think the choice boils down to 50:50 is another example of cognitive bias – it’s the opposite of the gambler’s fallacy where someone predicting the result of an 11th coin toss where the first ten have all been heads, will generally vote tails as they feel it’s overdue. In reality, the coin is either biased or the chances remain 50:50 – unlike our example above, the previous result has no bearing on future results.

For testers, awareness of these cognitive biases is important. As far as possible, we need to appreciate the actual expected results of our tests, without bias.

One place to go to get this awareness is John Stevenson’s blog, which often looks at psychology and its impact on testing.

Congratulations, you’ve won!

2 Comments

Congratulations, you’ve won!

Consider this, you’ve just won a quiz show and the chance of a million pounds (or the equivalent in your local currency). There’s just one catch, you have to guess which of three doors the cash is hiding behind.

The quiz show host takes you over to the doors and makes a big song and dance about how there’s a million pounds behind one of the doors, but tins of beans behind the two losing ones. Unless you’re a big fan of beans, I’m guessing you’re still trying for the million pounds.

It’s now crunch time and you’ve got to make a decision. This bit is easy as you’ve effectively got a one-in-three chance, unless you believe that fate, luck or something else is guiding your decision. Let’s assume that you pick door number one.

Here comes the twist. The quiz show host now explains that at least one of the remaining two doors must conceal a tin of beans, and he’s going to open that door. With great fanfare, he opens door number two, to reveal a fairly large tin of beans. He then gives you the chance to change your mind. Do you want to stick with your first choice, your gut instinct – or do you want to switch to door number three?

The quiz show host knows which door the million pounds is behind, but his face isn’t giving anything away. Your choice is to stick, or move. What choice do you make?

(I’ll reveal the answer and the reasons in the next blog post, unless anyone fancies trying to beat me to it in the comments)

Go the extra mile

Leave a comment

In my roles, wherever I’ve worked, I’ve often been asked to test a new piece of functionality that had only vague requirements and came with little in the way of instruction. In addition to testing it, I also had to work out what the process was either by asking people, or by good old fashioned trial and error.

I’d usually then document that process for myself, either for test scripts, or as a cheat sheet to remind myself what was the next step, after I come back from documenting a defect.

For quite a few of these, I would then find myself co-ordinating UAT; sitting with the user and walking them through the process to determine if what they’ve got at the endpoint matches up with what they wanted in the first place.

This user isn’t usually the only person to be using the new functionality, and they’ll end up going back and going through it all with their team.

What struck me after going through this a couple of times was that I’ve got the process there, documented – and this would be of real benefit as a training aid. It’s not usually especially user-friendly in the form of a test script or cheat sheet, but spending just an extra 20 minutes tidying up the formatting and adding a few screenshots, and you’ve suddenly got an attractive one or two sheet training document that can be handed to the user along with the new code.

Most of the work was being done anyway, so you’re not doing much extra, and I’ve found that the users really appreciate it. I’ve mentioned before about adding value and contributing to the overall solution – this is just another example of how a tester can bring more to the table than might be immediately apparent.

What Could Possibly Go Wrong?

2 Comments

Oh, how many times have I seen this mistake made? Probably about as many times as I’ve made it myself!
It’s all too easy to do, this is ‘just an upgrade’ or a ‘straightforward change’ or even a ‘simple addition’. The testing is therefore assumed to be minimal, and to take very little time as everything is bound to pass.

Of course, what actually happens is the same for most IT projects – the scope was underestimated and the finished product has more bugs than we thought it would. What was scheduled to take a day is suddenly lasting a week, developers who are needed to fix the bugs have already had their time assigned to other projects, and deadlines are either fast approaching or have already been and gone.

There’s no real remedy to this that I know of outside of pessimism from the outset – this pessimism leads to increased estimates and contingency, although there’s no guarantee that the project will agree to either! Assume that the application you’re going to get will fall over as soon as you even glance at it. Assume that everything will collapse with even the most basic of tests – that way you can only ever be pleasantly surprised.

Older Entries