Yesterday I was asked to add some kind of reporting to our selenium test suite. I had some ideas to create a report in excel, but this all ended up in having to write a lot of code and probably reinventing the wheel. After some time using google I came across the website of Bas Dijkstra where he was reviewing ExtentReport version 1.4. Before reading the rest of my post you might want to read this post. After reading I became pretty exited about this library! Next I went to see the website of the creator of ExtentReports, ReleventCodes.com. I installed the latest version (at the time 2.04) by adding it to my maven pom.xml and gave it a go. Below my approach and some tips.

Installation:

For installation I used maven. Add the following to the pom.xml of your test project:

<dependency>
    <groupId>com.relevantcodes</groupId>
    <artifactId>extentreports</artifactId>
    <version>2.04</version>
</dependency>

Open your command prompt and navigate to where ever your pom.xml is located and execute the following command:

mvn install -DskipTests
And your ready to go!

Using Junit? Use a @Rule annotation!

Since we already developed a “complete” automation test suite I didn’t want to add a new reporting line after every test we have written. When your experimenting with this library you could do this but later on you probably want to put this on a higher level in your testsuite.
First of all I want to tell you a little about the testsuite. I’m using Selenium webdriver (Java) and I developed our suite using the PageObjectPattern. I wrote a baseTestClass and all other tests extend this class. The baseTestClass responsibility is to setup Selenium (start a browser, navigating to a URL, truncate a database, etc.) I wrote a screenshotTaker class earlier which makes a screenshot of the browser when the test fails. This screenshotTakerClass has been added as a rule to the baseTestClass. This seems like a good starting point for me to see if a test fails if it creates a report and how it looks. For now I just added a line of code to the “failed” method of this specific class:

BaseTestClass snippet:
See line 11 for the rule where we call on the screenshotTakerClass

public class BaseTestclass {

	public static WebDriver driver;	
	/*
	 * Set some properties as a member.
	*/	
	private static String screenshotLocatie;
	private static String browser;
	
	@Rule
    public ScreenshotTaker screenShootRule = new ScreenshotTaker(driver, screenshotLocatie);

	@BeforeClass
	public static void setup() throws Exception {
		getProperties();
		
		//empty the database before testing.
		resetDatabase();
		if ("Firefox".equals(browser)) {
			driver = new FirefoxDriver();
		} else if("InternetExplorer".equals(browser)) {
			driver = new InternetExplorerDriver();
		} else {
			//fallback when browser propertie is not set to firefox.
			driver = new FirefoxDriver();			
		}
		//set the browser to max size
		driver.manage().window().maximize();
	}
}

screenshotTakerClass:

public class ScreenshotTaker extends TestWatcher {
    private WebDriver browser;
    private String screenshotLocation;
	private String filenameOfReport= "C:/myReportingDir/testReport" + DateTimeHelper.getDateOfToday() + ".html";
    
    public ScreenshotTaker(WebDriver browser, String screenshotLocatie) {
        this.browser =  browser;
        this.screenshotLocation =  screenshotLocation;
    }

    @Override
    protected void failed(Throwable e, Description description) {
        TakesScreenshot takesScreenshot = (TakesScreenshot) browser;

        File scrFile = takesScreenshot.getScreenshotAs(OutputType.FILE);
        File destFile = getDestinationFile(description);
        try {
            FileUtils.copyFile(scrFile, destFile);
        } catch (IOException ioe) {
            throw new RuntimeException(ioe);
        }
        ExtentReports extent = createReport();
        ExtentTest test = extent.startTest(description.getDisplayName(), "Test failed, click here for further details");

		// step log
		test.log(LogStatus.FAIL, "Failure trace Selenium: "+ e.toString());
		flushReports(extent, test);
    }
    
    //When passed only write to the log.
    @Override
    protected void succeeded(Description description) {
		ExtentReports extent = createReport();
         ExtentTest test = extent.startTest(description.getDisplayName(), "-");

 		// step log
 		test.log(LogStatus.PASS, "-");
 		flushReports(extent, test);
    }

    private ExtentReports createReport() {
    	ExtentReports extent = new ExtentReports(filenameOfReport, false);
        extent.config().reportName("My first extentReport report");
        extent.config().reportHeadline("See my awesome passed tests!");
        return extent;
    }
    
    private void flushReports(ExtentReports extent, ExtentTest test){
    	// ending test
 		extent.endTest(test);
 		// writing everything to document
 		extent.flush();
    }
    
    private File getDestinationFile(Description description) {
        String userDirectory = screenshotLocation;
        String date = getDateTime();
        String fileName = description.getDisplayName() + "_" + date + ".png";
        //add date of today
        String dateForDir = DateTimeHelper.getDateOfToday();
        String absoluteFileName = userDirectory + "/" + dateForDir + "/" + fileName;

        return new File(absoluteFileName);
    }
    
    private String getDateTime() {
    	Date date = Calendar.getInstance().getTime();

        // Display a date in day, month, year format
        DateFormat formatter = new SimpleDateFormat("dd-MM-yyyy_HH_mm_ss");
        String today = formatter.format(date);
        return today;
    }
}

After adding this to existing screenshotTakerClass I let JUnit run a complete test of our testsuite.
Every single test we have is added to the report, passed or failed!

Because I do not have to add a log line after every test I have to pass on the name of the test to identify the failing (or passed) test.
I came to this line (see line 23 and 34 in the code above)

ExtentTest test = extent.startTest(description.getDisplayName(), "Description of the test here....");

In the report it will show up as methodName(packageName) (for example: myAwsomeTest(com.mytestproject.seleniumtests.mostawesometests).

Next thing I wanted is to show the failure trace to see what went wrong:
I added this line:

test.log(LogStatus.FAIL, "Failure trace Selenium: "+ e.toString());

This one only applies to the failure, when a test is passed I don’t need any more information then it is passed:

test.log(LogStatus.PASS, "-");

After experimenting a little bit with ExtentReports I changed the reporting name to make it more suitable to the test project.

extent.config().reportName("Selenium - Awesome project");
extent.config().reportHeadline("Overview of testresults");

That’s it for now! After this modifications the report suited just fine for me. Only thing left is to rename the screenshotTakerClass since it now has more responsibilities.
I think EventReports is a very nice library to use when you have the need to build reports for Selenium. It will probably also fit other Java based test tools.

For more detailed information and other possibilities in Extend reports see:
http://extentreports.relevantcodes.com/2x/docs.html

Questions? Please leave a comment below!

Exploratory testing.

Not a long time ago I had a discussion with someone from my organization about testing and the way we test our product at the organization I work for. We had an almost endless discussion about what exploratory testing is and why we practice this kind of testing. I was shocked to hear him say that: “Exploratory testing should be a tester’s only thing”. From my perspective this doesn’t have to be tester’s only, testers could surely use the input of consultants and / or developers when exploratory testing. And when needed perhaps we should test together! I got the feeling that he had no clue about what Exploratory testing really means and what it could mean for the stability of our product. Perhaps you have experienced the same discussion, or experiencing it right now. So here is what I think about exploratory testing.

So what is exploratory testing?

Let’s start off with why we do exploratory testing. Many of you reading this article will probably have heard of the term “Exploratory testing” before since there is already written so many about this topic. Many testers already apply this way of testing  for some time.

In short, Exploratory testing is testing a system under test (or a part of it) without following a strict test script. Take a part of the system under test where multiple components come together and test the interaction of these components. It will give you information about how the system works and if the interaction between components is correct and understandable for the (most of the times less technical) end user. So you will be designing and executing test cases on the spot. Use your curiosity to experiment with the components and see if what ever happens is as you expected it to.

By exploring the functionality you probably execute existing test cases (happy path) but you will also test paths not covered in test cases (perhaps resulting in new (automated) test cases to be executed). The more you know about the product that is developed and the more you are experienced with testing, the better the result of the exploratory test.

Is exploratory testing only for testers?

From my perspective I would say: No, not totally! Perhaps a exploratory test is at best when executed by a tester since our testing skills and mindset would help when testing. But like I mentioned before: The more you know about the product that is developed…. That would indicate that everyone from developers to consultants could be involved in exploratory testing. Not only testers have knowledge about the product developed. For example: Consultants usually know how one or more customers use the product. This knowledge can be of great value when they are testing. Not all developers and tester will have this knowledge.

Developers for example know how they developed the product and where to find “weak” spots in the application. They can point them out for you to test, but can also test these themselves, learn, fix bugs and deliver a better (part of the) product. Developers can also practice exploratory testing when doing a code review. Besides literally only reviewing the code, click trough the component if possible and look for some simple bugs. By saying that I’m not saying that testers are obsolete, I think testers just shouldn’t be bothered with finding bugs that are blocking the flow just after two mouse clicks.

Why exploratory testing is so important:

Following test scripts is repetitive work, testing a new release for the second time by following a testscript will probably not give you new results. It’s most unlikely that these test will now suddenly fail. By exploring (parts of) the system under test you will probably find new results, find new testcases, write them down and perhaps automate them. The first thing I do when I’m done with testing a new version of our application via a manual regression testscript is exploratory testing. This can also be done when your in an agile team and the user store is completed because it reached its definition of done, after that there is nothing holding you back to explore the functionality and put it to done when you as tester are completely satisfied with the result.

How to overcome some of the downsides of Exploratory testing

Unfortunately exploratory testing also has some downsides. The main problem I experienced was that you can get easily lost in time when exploratory testing, you’ll have to watch out that it down not result in some kind of exhaustive testing. Best practice for me is to time box each exploratory testing session. For each time frame I pick one or two (when testing interaction) particular parts of the system under test, set some objectives what to test, write down or log my findings and when the time is up I move on to the next test.

An easy and lightweight tool to help you with creating sessions for exploratory testing is Sessiontester. As described by their website: “It has a timer so you limit your test sessions to a particular length, and it provides an easy way to record session notes.”

When to apply exploratory testing

Here is a summary of situations in which you could prefer applying exploratory testing:

• When testing with test scripts is completed but you do not feel totally confidant with the delivered functionality.
• When fast feedback about a functionality is needed.
• When the functionality is already mature and no or minor changes has been made.
• When there is no time to execute a (scripted) functional test but feedback is still required.
• When testing the frontend of a webapplication with multiple browsers (Internet explorer, Firefox, Safari etc.).

Some interesting links:

James bach about exploraty testing.
A PPT about Exploratory testing in Agile by Elisabeth Hendrikson
The sience of Exploratory testing