Hamcrest Test Framework

 Hamcrest란?

test framework을 이용하여 단위 테스트 진행 시 assert를 유연하게 하기 위해 matcher를 이용하여 좀 더 쉽고 확장성 있는 assert를 하기 위해 만들어진 framework이다. 

Hamcrest는 처음부터 test framework과 통합되도록 설계되어 JUnit, TestNG와 함께 사용할 수 있다.

maven dependency

<dependency>
    <groupId>org.hamcrest</groupId>
    <artifactId>hamcrest-library</artifactId>
    <version>2.2</version>
    <scope>test</scope>
</dependency>

Hamcrest code

assertThat(actual, is(equalTo(expedted);
assertThat(result, instanceof String);
assertThat(result, is(hasItem(anyOf(equalTo("x", equalTo("y"), equalTo("z"))))));

Hamcrest Matcher 

allOf - matches if all matchers match (short circuits)

anyOf - matches if any matchers match (short circuits)

not - matches if the wrapped matcher doesn’t match and vice

equalTo - test object equality using the equals method

is - decorator for equalTo to improve readability

hasToString - test Object.toString

instanceOf, isCompatibleType - test type

notNullValue, nullValue - test for null

sameInstance - test object identity

hasEntry, hasKey, hasValue - test a map contains an entry, key or value

hasItem, hasItems - test a collection contains elements

hasItemInArray - test an array contains an element

closeTo - test floating point values are close to a given value

greaterThan, greaterThanOrEqualTo, lessThan, lessThanOrEqualTo

equalToIgnoringCase - test string equality ignoring case

equalToIgnoringWhiteSpace - test string equality ignoring differences in runs of whitespace

containsString, endsWith, startsWith - test string matching

TestNG의 ReportNG 설정법

pom.xml

<dependencies>
    <dependency>
        <groupId>org.seleniumhq.selenium</groupId>
        <artifactId>selenium-java</artifactId>
        <version>${selenium.java.version}</version>
    </dependency>
    <dependency>
        <groupId>org.uncommons</groupId>
        <artifactId>reportng</artifactId>
        <version>${reportng.version}</version>
    </dependency>
    <dependency>
        <groupId>com.google.inject</groupId>
        <artifactId>guice</artifactId>
        <version>${guice.version}</version>
    </dependency>
    ......
</dependencies>
<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>${maven.surefire.version}</version>
            <configuration>
                <encoding>UTF-8</encoding>
                <argLine>-Dfile.encoding=UTF-8</argLine>
                <properties>
                    <!-- Setting ReportNG listeners -->
                    <property>
                        <name>usedefaultlisteners</name>
                        <value>false</value>
                    </property>
                    <property>
                        <name>listener</name>
                        <value>org.uncommons.reportng.HTMLReporter,
                            org.uncommons.reportng.JUnitXMLReporter</value>
                    </property>
                </properties>
                <suiteXmlFiles>
                    <suiteXmlFile>src/test/resources/testng.xml</suiteXmlFile>
                </suiteXmlFiles>
            </configuration>
        </plugin>
        ......
    </plugins>
</build>

testng.xml(test-suite)

<!DOCTYPE suite SYSTEM "http://beust.com/testng/testng-1.0.dtd" >
<suite name="TestNG Suite Name">
    <listeners>
        <listener class-name="org.uncommons.reportng.HTMLReporter" />
        <listener class-name="org.uncommons.reportng.JUnitXMLReporter" />
    </listeners>
    <test name="tmon_monthly_maintenance">
        <classes>
            <class name="com.tmoncorp.test.maintenance.TmonHomeTest" />
            <class name="com.tmoncorp.test.maintenance.UserPageTest" />
            .....
        </classes>
    </test>
</suite>

TestNG에 ExtentReport 적용하기

1. pom.xml dependency 추가

<dependency>    
  <groupId>com.aventstack</groupId>    
  <artifactId>extentreports</artifactId>    
  <version>3.1.5</version>
</dependency>

2. ExtentManager 추가

public class ExtentManager {
    private static ExtentReports extent;

    public static ExtentReports getInstance() {
        if (extent == null)
            createInstance();
        return extent;
    }

    //Create an extent report instance
    public static ExtentReports createInstance() {
        String reportFile = new File("target/result.html").getAbsolutePath();
        ExtentHtmlReporter htmlReporter = new ExtentHtmlReporter(reportFile);
        htmlReporter.setAppendExisting(true);
        htmlReporter.config().setTestViewChartLocation(ChartLocation.BOTTOM);
        htmlReporter.config().setChartVisibilityOnOpen(true);
        htmlReporter.config().setTheme(Theme.STANDARD);
        htmlReporter.config().setDocumentTitle(reportFile);
        htmlReporter.config().setEncoding("UTF-8");
        htmlReporter.config().setReportName(reportFile);

        extent = new ExtentReports();
        extent.attachReporter(htmlReporter);
        extent.setSystemInfo("Environment", "QA");
        TestData testData = new TestData();
        extent.setSystemInfo("Host Name", testData.getTestData().getUri());
        return extent;
    }
}

3. TestListener 추가

public class TestListner implements ITestListener {
    //Extent Report Declarations
    private static final Logger logger = LoggerFactory.getLogger(TestListner.class);
    private static ExtentReports extent = ExtentManager.createInstance();
    private static ThreadLocal<ExtentTest> test = new ThreadLocal<>();

    @Override
    public synchronized void onStart(ITestContext context) {
        logger.debug("Test Suite started!");
    }

    @Override
    public synchronized void onFinish(ITestContext context) {
        logger.debug("Test Suite is finished!");
        extent.flush();
    }

    @Override
    public synchronized void onTestStart(ITestResult result) {
        logger.debug(result.getMethod().getMethodName() + " started!");
        ExtentTest extentTest = extent.createTest(result.getMethod().getMethodName(),result.getMethod().getDescription());
        extentTest.assignCategory(result.getTestClass().getRealClass().getSimpleName());
        test.set(extentTest);
    }

    @Override
    public synchronized void onTestSuccess(ITestResult result) {
        logger.debug(result.getMethod().getMethodName() + " passed!");
        test.get().pass("Test passed");
    }

    @Override
    public synchronized void onTestFailure(ITestResult result) {
        logger.debug(result.getMethod().getMethodName() + " failed!");
        test.get().fail(result.getThrowable());
    }

    @Override
    public synchronized void onTestSkipped(ITestResult result) {
        logger.debug(result.getMethod().getMethodName() + " skipped!");
        test.get().skip(result.getThrowable());
    }

    @Override
    public void onTestFailedButWithinSuccessPercentage(ITestResult result) {
        logger.debug("onTestFailedButWithinSuccessPercentage for " + result.getMethod().getMethodName());
    }
}

4. testng.xml에 추가

<suite name="TestNG Suite Name">
    <listeners>
        <listener class-name="com.xxxx.utils.listners.TestListner" />    
    </listeners>
    <test name="full_test">
         <packages>
             <package name="com.xxx.test.*" />
         </packages>
     </test>
</suite>

TestNG에서 테스트 순서 제어하기

public class TestNGTest3 {
 
    @BeforeClass
    public void beforeclass(){
        System.out.println("@BeforeClass TestNGTest3");
    }
     
    @Test
    public void test7(){
        System.out.println("test7");
        Assert.assertTrue(true);
    }
     
    @Test
    public void test2() throws Exception{
        System.out.println("test2");
        Assert.assertTrue(true);
    }
     
    @Test
    public void test15() throws Exception{
        System.out.println("test15");
        Assert.assertTrue(true);
    }
     
    @AfterClass
    public void afterclass(){
        System.out.println("@AfterClass TestNGTest3");
    }
}

[결과]

@BeforeClass TestNGTest3
test15
test2
test7
@AfterClass TestNGTest3
PASSED: test15
PASSED: test2
PASSED: test7

public class TestNGTest1 {

    @BeforeClass

    public void beforeclass(){
        System.out.println("@BeforeClass TestNGTest1");
    }

    @Test
    public void test12() throws Exception{
        System.out.println("Test12");
        Assert.assertTrue(true);
    }

    @Test
    public void test5(){
        System.out.println("Test5");
        Assert.assertTrue(true);
    }

    @Test
    public void test3(){
        System.out.println("Test3");
        Assert.assertTrue(true);
    }

    @AfterClass
    public void afterclass(){
        System.out.println("@AfterClass TestNGTest1");
    }
}

[결과]

@BeforeClass TestNGTest1
Test12
Test3
Test5
@AfterClass TestNGTest1
PASSED: test12
PASSED: test3
PASSED: test5

[해결책]

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="TestNG Suite Name">
    <test name="TestNGTest" preserve-order="true">
        <classes>
            <class name="testng.TestNGTest1">
                <methods>
                    <include name="test3" />
                    <include name="test5" />
                    <include name="test12" />
                </methods>
            </class>

            <class name="testng.TestNGTest3">
                <methods>
                    <include name="test2" />
                    <include name="test7" />
                    <include name="test15" />
                </methods>
            </class>
        </classes>
    </test>
</suite>

[설명]
testng의 suite 파일에 preserve-order='true' 를 설정하고 해당 클래스의 test method의 순서를 지정해주면 된다.

[결과]
@BeforeClass TestNGTest1
Test3
Test5
Test12
@AfterClass TestNGTest1
@BeforeClass TestNGTest3
test2
test7
test15
@AfterClass TestNGTest3

※ 위 순서대로 사용하려면, dependsOnMethod를 사용하시면 안됩니다.

TestNG에서 2개 이상의 assert를 한꺼번에 처리하는 방법

개요

"@Test" 메소드에서 테스트 할 때 부득이하게 assert를 여러 번 해야 할 경우가 있다.

이 때 한꺼번에 처리해주지 않으면 첫번 째 assert에서 fail이 발생할 경우, 그 아래에 있는 assert 구문을 처리하지 않고 테스트가 종료되어 남아있는 assert 구문을 확인할 수 없다.

이를 해결하기 위한 방법을 정리한다.

내용

TestNG에서는 Hard assert / Soft Assert로 구분한다.

Hard Assert는 assert 시 fail이 발생하면 해당 테스트를 종료하는 것이고,

Soft assert는 assert 시 fail이 발생해도 로그에서는 fail이지만 테스트 결과는 pass로 해준다. 최종 결과를 확인하기 위해서는 아래와 같이 사용해야 한다.

Hard Assert

@Test
public void hardAssertTest(){
   Assert.assertFalse(2<1);
   System.out.println("Assertion Failed in Test 1");
   Assert.assertTrue(1<0);
   System.out.println("Assertion Failed in Test 2");
   Assert.assertEquals("Sample", "Sample");
   System.out.println("Assertion Passed in Test 3");
}

결과
Assertion Failed in Test 1 FAILED: hardAssertTest1 java.lang.AssertionError: expected [true] but found [false] at org.testng.Assert.fail(Assert.java:94) at org.testng.Assert.failNotEquals(Assert.java:513) at org.testng.Assert.assertTrue(Assert.java:42) at org.testng.Assert.assertTrue(Assert.java:52) at testng.TestNGTest2.hardAssertTest1(TestNGTest2.java:13) ...

3번 line : PASS, 5번 line : FAIL이 발생하여 7번line의 assert를 하지 않고 hardAssertTest()의 테스트를 종료하고 빠져나온다. 테스트의 최종결과는 FAIL

Soft Assert 1

@Test
public void softAssertTest1(){
   SoftAssert sa= new SoftAssert();
   sa.assertTrue(2<1);
   System.out.println("Assertion Failed1");
   sa.assertFalse(1<2);
   System.out.println("Assertion Failed2");
   sa.assertEquals("Sample", "Failed");
   System.out.println("Assertion Failed3");
}

결과
Assertion Failed1 Assertion Failed2 Assertion Failed3 PASSED: softAssertTest1

assert는 fail이지만 테스트 결과는 PASS로 된다.

Soft Assert 2

@Test
public void softAssertTest2(){
   SoftAssert sa= new SoftAssert();
   sa.assertTrue(2<1);
   System.out.println("Assertion Failed1");
   sa.assertFalse(1<2);
   System.out.println("Assertion Failed2");
   sa.assertEquals("Sample", "Failed");
   System.out.println("Assertion Failed3");
   sa.assertAll();
}

결과
Assertion Failed1 Assertion Failed2 Assertion Failed3 FAILED: softAssertTest2 java.lang.AssertionError: The following asserts failed: expected [true] but found [false], expected [false] but found [true], expected [Failed] but found [Sample] at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:43) at testng.TestNGTest2.softAssertTest2(TestNGTest2.java:54) ....

SoftAssert 객체를 생성한 후 - SoftAssert sa = new SoftAssert();

SoftAssert 객체로 assert를 모두 진행하고 난 뒤, 마지막에 "sa.assertAll()" 을 해주면 하나라도 fail이 발생 시 최종 테스트 결과는 FAIL로 된다.

assert를 모두 확인하고 테스트를 종료하므로 테스트 메소드에 assert 구문이 여러 개 일 경우 SoftAssert의 assertAll을 사용하자.