What Could Possibly Go Wrong?


Oh, how many times have I seen this mistake made? Probably about as many times as I’ve made it myself!
It’s all too easy to do, this is ‘just an upgrade’ or a ‘straightforward change’ or even a ‘simple addition’. The testing is therefore assumed to be minimal, and to take very little time as everything is bound to pass.

Of course, what actually happens is the same for most IT projects – the scope was underestimated and the finished product has more bugs than we thought it would. What was scheduled to take a day is suddenly lasting a week, developers who are needed to fix the bugs have already had their time assigned to other projects, and deadlines are either fast approaching or have already been and gone.

There’s no real remedy to this that I know of outside of pessimism from the outset – this pessimism leads to increased estimates and contingency, although there’s no guarantee that the project will agree to either! Assume that the application you’re going to get will fall over as soon as you even glance at it. Assume that everything will collapse with even the most basic of tests – that way you can only ever be pleasantly surprised.


Beautiful Scripts


Most testers I’ve seen just write test scripts without ever going near the formatting options – this is such a waste!
Consider the below two examples:

1. Login to application
2. Access Subscriptions form
3. Add a subscription with Name=”Test Subscription AD01″, Start date=”01/01/11″, End date = “31/12/15”
4. Save record
5. Close the form

1. Login to application

2. Access Subscriptions form

3. Add a subscription with
Name=”Test Subscription AD01“,
Start date=”01/01/11“,
End date = “31/12/15

4. Save record

5. Close the form

Which of the following is easier to read? Imagine that simple test scaled up to a 30-40 step test, and how much easier to interpret it would become with formatting.

Even Excel allows for different formatting within the same cell, so there’s no excuse not to do it.
Choose your own standard, but I tend to make names of forms, windows columns, fields etc. bold. Key actions (e.g. Login) are also bolded. The actual data is marked bold and red.

Of course, I’m not advocating spending hours needlessly turning your scripts into a work of art, but adding some basic formatting will improve readability and acts as a review period, when you can step back for a second and look at what you’ve just written.

Why Write Scripts?

Leave a comment

How detailed should test scripts be?

At the start of any project you would normally think at a high level about the approach you were going to take, and possibly present this in a plan. At some point you’ll get that down to a finer level of detail, thinking about individual scenarios or processes to look at. Beyond that you have permutations and then beyond that you have your test steps (and in most cases, expected results).

Let’s take an example – a new form has been created that holds subscriptions and the associated from/to dates.

Our plan would be along the lines of creating new subscriptions, then editing and deleting existing ones. We would also intend to look at date range validity.

We’d then look at scenarios:

  • Add record
  • Modify record
  • Delete record
  • Check for valid dates
  • Check for start date before end date
  • Check for start date same as end date
  • Check for non valid date types/strings/symbols etc.
  • Do each of the above checks for new records and then edits

The list goes on…

You can then look at permutations, which are essentially the above, but with actual data in.

You then, finally get down to the test steps:

1. Login to application

2. Access Subscriptions form

3. Add a subscription with:

Name=”Test Subscription AD01″, Start date=”01/01/11″, End date = “31/12/15”

4. Save record

5. Close the form

Now, the above steps would actually hold a lot more detail, such as how we access that subscription form – is it directly, or is it attached to a customer? If it’s directly, then how do we associate the subscription with a customer? All that detail though, as well as steps 1, 2 and 4, remain the same in almost every scenario we’ve posited.

I’ve seen some arguments that suggest that we don’t need to go to that level of detail – a competent tester can work from scenarios or at most, the permutations.

I’ve seen other arguments that insist that full steps must be written out for each test. You’ve no guarantee who will be running that test, and it’s your job to write it such that the proverbial ‘man on the street’ can run it.

I’ve actually been told this one so often that I started to believe it, despite the fact that it cheapens our role. The biggest problem with this approach is that it is tantamount to writing an automated test script, and has most of the disadvantages that come with that, while having few of the advantages, as the script is still to be run by a human.

I take a different approach.

I believe that full steps for the process are necessary. Our example is a completely new form – what if the link has been put in the wrong place, or if the link once found actually opens up the wrong form? What if saving the record or closing the form gives an error? Essentially, by eliminating steps you are introducing assumptions – and none of those assumptions are guaranteed to be true.

So, by writing the full steps we are forced to discard all assumptions. We’ve got to state upfront where the link for the form will be, and in the absence of a working application, we’ll have to rely on documentation – so we’re now effectively testing the documentation for completeness as we write our scripts.

That being said, with all the scenarios given, there’s a huge amount of repetition, and although computers make it easy to copy and paste it’s wasteful to duplicate all the steps for each one.

I believe that we should start out being as complete as possible, and then for subsequent tests we can gradually drop the detail.

Using the example above, we keep what we’ve got for test one, but test two becomes:

1. Add a new subscription:

Name = “Test subscription AD02”

Start Date = “31/12/2015”

End Date = “01/01/2011”

2. Save the form – note that an error is given, indicating that the end date is before the start date.

This way, we’ve struck a balance between completeness and efficiency. We’ve made no assumptions as our process is fully documented, but we haven’t endlessly and needlessly repeated ourselves.

Most importantly it allows us to add further ideas for tests quickly and easily. No amount of documentation is a substitute for getting your hands on the application, and once you’ve got some experience of actually using the application you’re testing, you’ll come up with more ideas inspired by how the UI works, what sort of responses you’re getting, defects you’ve raised etc.

Not all of this exploratory testing will be documented, but it’s far less likely to be documented at all if every instance of a test has to have a full wrapper of basic starting steps.

But I Didn’t Order Vanilla! A Rant…

Leave a comment

I’m fed up of being told that a bug or usability issue I’ve found is ‘vanilla functionality’.

Wherever I’ve worked, when this situation has cropped up, the only reason development has been done and testing is needed, is because we didn’t order vanilla!

For those not familiar with the jargon, the term vanilla refers to plain and unaltered, the ‘original’ state of the software before your particular flavour of enhancements and modifications are added.

An example would be some generic software that creates invoices. The first change you’d be likely to make would be to add your own company logo. The rest of the software would then be vanilla. Later on you might want to add a facility to specify a discount, say 5% to apply to resellers.

Now, say I’m testing this software. My focus is on the logo and the presence of the reseller discount when appropriate. I’m not a computer running an automated test though, and if I spot a typo on the terms and conditions, or that the VAT rate isn’t right, then I’m going to flag it up as a defect.

On one hand, I can understand and appreciate the developers’ response; this has nothing to do with the change and so they can’t be held responsible for it. On the other hand, we’re all trying to produce the best work we can, in a usually hyper-competitive environment, so just occasionally I’d like the response to be ‘OK, let me see what I can do’.

We know that the flavour needed to be customised, that our company wanted strawberry ice cream as vanilla didn’t quite cut it. But if, when we taste the results of the strawberry recipe, the flavour is too sharp, we shouldn’t be against the general principle of adding a little more sugar.

One Quadrillion dollars!

Leave a comment

It’s more than Dr Evil ever dared ask, but GRR Airport has no such qualms…

Over the weekend I noticed twitter was buzzing, and in particular I noticed a conversation between @testertested and @jamesmarcusbach in which they were discussing an ever increasing total for car parking. Intrigued, I asked for more info and ended up joining in with testing the boundaries of a car parking calculator for Gerald R Ford International Airport. Go try it for yourself – see what the highest value car parking charge you can get is. As a starter, I’ll tell you that the record is in the quadrillions of dollars (that’s a one with 15 zeroes on the end).


 I started off gently, getting the charges for 1/1/2000 to 1/1/2030, then going through each of the available types of parking to see which was the most expensive. My first assumption to be challenged was that valet parking would be the most expensive when it turned out that as costs increased, short-term parking was far more expensive and indeed continued to be so until the end.

I then ramped up the end date to 31/12/9999 and watched the figures climb, but they were still nowhere near the numbers I’d seen on twitter, so I carried on.

 My next thought was to set the start date to BC, or rather to enter in a negative value, so I reversed my ‘to’ value and entered in 1/1/-9999 and sure enough this worked, but I was still a way away from the numbers I’d seen achieved by others, so I persevered. This time, I tried adding more 9s to the year, and here was when I ran into my first validation error. Although the field did not restrict how many characters I could add, at a certain point the calculator said that my end date was before my start date. Now that I’d finally exceeded the limit I could fine-tune my numbers to find out what it was. I got to a stage with a certain number of 9s in the year field, and had a charge for that period BC to that period AD. I set the times to 00:00 to 23:59 and looked at my ‘score’ of over $1.8 quadrillion, and was reasonably happy. I tweeted the result – soon after I got the reply:

 “That is about a third of the record”

 A third??! That spurred me on. I quickly realised that I was being too kind to the application and not testing it as I normally would. If the application allowed negative values for the year, then why not the month and the day? Entering -31st of the -12th month improved my score further, and similarly I was able to add negatives in the time field. It then occurred to me that I was sticking to traditional day and month values needlessly, the application obviously wasn’t doing much validation as it was allowing negatives.

 I applied my ‘year’ value to my day and month, and this worked. I tried increasing it, but as with the year this caused a validation error. I then applied the same numbers to my time field, again with positive results and an increasing score. I was now getting closer, but still not close enough. It was then that I realised my second incorrect assumption. I’d assumed that the limitation on the day, month, year etc. was in the number of characters; however I found that I was able to increase the absolute number by one, successfully.

I continued increasing and realised that I could keep all of the 9s that I had, and then add a 7 at the front. Once I replaced all my numbers with this seemingly limiting value, then my score was much higher, at $4.985 quadrillion. The record is $5.124 quadrillion, so I was within 4%.

 At this point I’d used up my half hour, which was the limit that most people seemed to be using, so I stopped, reasonably content. Over the next couple of days I had a few more thoughts about it, and again realised that my biggest mistake was in being too kind to the application and playing by rules that it wasn’t necessarily following, e.g.

  • I was putting in a day, month and year – I hadn’t tried with more or less than three values.
  • I was putting in a time – the time may have had too many characters, but there was still a ‘:’ in the middle
  • I had restricted myself to numbers – which I wouldn’t have done if I was trying to break the app rather than achieving a high score, however it turned out that the letter ‘e’ is present in the parameters for the best score.

 Thanks to @jamesmarcusbach and @testertested for the inspiration and link, and for those of you that want to see the high score, here’s the link: http://bit.ly/faltVH

Getting Involved Early

Leave a comment

I recently read a blog here talking about the need to get testing involved earlier. I was particularly taken with this quote:

“Good test managers try to accentuate the advantages of early participation and working at the same time with development…”

Is this solely the preserve of the Test Manager, or can any tester accentuate these advantages – and just what are they anyway? As we seek to improve our craft we all need to think about what we’re doing, how we can do it better, and most importantly, what we need in order to achieve those aims. It may be that this needs to be escalated to the Test Manager, but often problems can be solved and improvements suggested much earlier on and without escalation.

Before we get onto the advantages, let me briefly outline what normally happens.

Testing is usually the final stage of the process before the users get their hands on the application, whether that is by going live, or by UAT. Ideally, testing and development should have proceeded so well that very few issues will be found (of course no application can ever be certified as bug free as complete testing is impossible)

As the final stage of the process, there are numerous associated issues.

First and foremost is that projects often have inflexible end-dates, and so as a project hits any delays, testing is the obvious candidate for being shortened – and if no phase is explicitly shortened, then testing is hit implicitly. That’s a well-discussed topic for another day though.

Second, is that testers are often late to the party. Documentation testing isn’t considered a part of the process, and the time devoted to gathering requirements, designing code followed by the actual development of that code is considered to be a lot of time for testers to be sat around ‘doing nothing’. The argument goes that testers can get involved later when all the documentation is finalised, or after development has started.

The issue with this is that most developers do not know how to test that requirements and design are accurate and complete, and that there’s no independent observer to cross-reference the two and ensure that the requirements are properly and fully covered by the design.

So, to try and mitigate against some of these issues, there is a concerted move to try and get the tester involved earlier. The benefits include:

a)      Testers by nature have an eye for detail. Getting them involved in reviewing requirement and design documentation is more likely to expose issues and get them addressed before they ever make it into code.

b)      If the testers are involved when the requirements are being written, then they’re likely to better understand them, leading to fewer questions later on, and better scripts written more quickly.

c)       By reviewing both the requirements documentation and the design documentation, testers are uniquely placed to ensure that the two line up, and that interpretations are consistent.

When presenting these points, draw on experience where possible too. I’ve written before about two developers next to each other coding a new ‘Priority’ field to be applied to two applications. One went for the values Low, Medium, High and Critical and the other went for 1, 2, 3 and 4. A lack of communication and reading each others’ work led to this error, but a tester working with them before the designs are committed to code, would have picked this up too.

Getting back to the original quote, these may well be points that a good Test Manager would make, but there’s no reason that any tester can’t suggest them initially. Remember that we’re not just trying to find bugs, but to contribute to the overall solution and increase the value that we add. So go on, think about what improvements you could make by being part of the project earlier on, then push for it to happen!

Ugly Babies

1 Comment

One of the earliest phrases I learnt in testing, and one of my favourites is that nobody likes being told that their baby is ugly. To put it another way, you’re going to spend a lot of your time as a tester criticising others’ work and you’re going to have to find a way to do that, that doesn’t just end up ticking them off.

There are two main areas where you’re going to come across this. The first is when you’re raising defects. I’ve blogged earlier about picking on every minute detail and raising defects for every ‘i’ not dotted, but it’s worth considering here that raising a hundred defects for typos and spelling mistakes isn’t going to win you any friends. These defects are going to show up on reports and are going to look bad, when the impact of them is probably minimal, and the time taken to fix them is likely to be minimal too.

I usually try and solve the problem by grouping these into one defect, or several defects grouped by page/section/functional area. This way you still get everything reported but you’re not repeatedly distracting the developer and project manager with what they see as minor issues.

Another technique (that can be combined with the above) is to do more than just raise the defect, by offering a suggestion on how to fix it. For spellings and typos it’s usually easy to supply the correct version as well. Be very careful here though not to be seen to be dictating a particular solution. Be clear that the solution is a potential one, or quote from the original specification to back up your claim.

Finally, ensure that you have a ‘Rejected’ option on your defect status list. We’re just as fallible as those that write the specifications and the code, and it’s to be expected that a proportion of the defects we raise will be invalid due to misunderstandings or not having access to all the information. As a counter to that, don’t be afraid to challenge a rejected defect, but accept that there will always be some.

Secondly is when reporting. You’re going to have graphs and charts that show failed tests and defects raised, and you’re going to have to comment on them. Choose your words carefully; assign blame to systems and applications rather than individuals and stick to the facts. Equally, emphasise the positives, and the efforts made to correct issues; this time ignore the systems and applications and focus on the individuals who got the defects fixed.

Have a separate section in your report for issues and recommendations. Identify the issues you came across, again avoiding assigning blame, and then provide suggestions on how those issues could be avoided in future. Crucially, always end on a positive note.

 The aim is to be seen as someone trying to contribute to the overall solution, not just someone who finds bugs.

Older Entries Newer Entries