Improve Automation Process with Tools and Patterns

Processes

The daily activities in an Test Automation team could be categories such as:

• Task Planning
• Development
• Syncing & status management

In the task planning stage, we collect task input from TestRail, examine them with playbook and create Jira ticket for development

In development stage, we automate the tests in code and make sure it passed in testing environment locally or on remote Team City server.

The syncing and status management happened during the above 2 stages, we need to keep the Jira, TestRail, GitHub synced with each other.

The process can be visualized as following:

    graph TD
    A(TestRail) -- Find Cases --> B
    B(Jira) -- Ticket Created --> C
    C(GitHub Branch) -- Push --> D
    D(Team City) -- Pass --> E
    D -- Not Pass --> C
    E(Github PR) -- Update Ticket --> B
    B -- Update Case Status --> A

Use this link if the flowchart isn't rendered properly.

Analysis

Among the process, the core and valuable part is merely the development ( GitHub <-> Team City loop ) which give us desired output . The other processes are supplement activities which can be reduced as less as possible.

Per this principle, there are points can be improved:

• Find Cases -> Use Analyzer Sheet
• Create Ticket -> Clone from template
• Sync Status -> Automation Scripts & Analyzer Sheet

The case finding process is directly affected by the quality of Test Rail case suite received as task input. This is the part that we can not control much.

Then once proper cases selected, bulk operation / automation can be introduced in:

• Create Jira tickets
• Update TestRail cases
• Sync TestRail cases per status of Jira ticket

Improvement

Automation

During the task go on, there might be lots of activities: TestRail cases need be updated as well as the Jira tickets.

So here the automation comes for help.

Take the "Jira sync back" as example, in this scenario, Jira ticket updated and we will need update TestRail cases accordingly.

What we need will be:

• TestRail case Ids as input
• Status of TestRail cases
• Status of Jira ticket created for cases

So here we can create a small script:

To get the status of TestRail cases:

def get_testrail_status(tr_case_id):
    tr_js = requests.get(
        test_rail_url_root + test_rail_get_case + tr_case_id,
        auth=(test_rail_usr, test_rail_pwd), headers=headers).json()

    case_id = str(tr_js['id'])
    case_title = tr_js['title']

    result = ''
    if tr_js['custom_tc_status'] == 3:
        result = ('Approved for Automation')
    if tr_js['custom_tc_status'] == 5:
        result = ('Automated')
    if tr_js['custom_tc_status'] == 7:
        result = ('Automation in progress')
    if tr_js['custom_tc_status'] == 9:
        result = ('Approved for Testing')
    return result

To get status of Jira tickets:

    pp = pprint.PrettyPrinter(indent=2)
    jira_url =  'https://jira.devfactory.com/rest/api/2/search?fields=key,summary,status' + \
                '&jql=project=%s AND type = "Integration Test" AND summary ~ "C%s"' % (jira_proj, tr_case_id)
    result = requests.get(jira_url, auth=(jira_usr, jira_pwd)).json()
    tr_status = get_testrail_status(tr_case_id)
    tr_status_fmt = '%-10s # %-25s @ '
    if len(result['issues']) > 0:
        print(tr_status_fmt % ( tr_case_id, tr_status) +
              '%-20s -> %s' % (result['issues'][0]['key'], result['issues'][0]['fields']['status']['name']))
    else:
        print(tr_status_fmt % ( tr_case_id, tr_status))
    return

Then we got output like:

Input cases:
1266040
1266043
1739712

Processing started...
1266040    # Automated                 @ VIEWSCID-15232       -> Closed
1266043    # Automation in progress    @ VIEWSCID-15438       -> Open
1739712    # Approved for Automation   @ 

Now we can see status from both TestRail and Jira accordingly. Full script can be found here

Avoid Conflict

There is also one important topic when team is working on a range of cases of same product. How would you determine the cases are not taken by one of your team mates? Use status of TestRail cases? Yes, but one powerful tool can be used as well for double check: JQL Queries

First make sure all create ticket following some kind of format pattern such as C[TestRail Case Id] TestRail Case Description.

Now assuming you are handling cases 12345 and it's Jira project name is EVCID, then query could be: project = EVCID AND type = "Integration Test" AND summary ~ "C1120561"

Patterns for UI Test Automation

The main pattern we are interested here is the Page-Object pattern. Though it's first designed for Webpage, but the principle is same for desktop UI automation.

The key of the pattern is modeling and model the UI element rather than steps.

In nature developer may implement one UI test in a sequence step by step literally based on the test cases definition. And then since D.R.Y principle is there, common used steps are extracted out.

The code may look like:

    var obj = new Class();
    obj.DoLoginAndFillLotsOfFields();
    AnotherClass.DoSomeKindOfProcedure(arg1, arg2)

This is fine but this is not all. The smallest item of test is the UI element rather than procedures. No matter how the procedure was modeled, there would be still repeat part when same dialog involved in different procedure. It may be find and operated on repeatedly in code for different procedure.

Another drawback of modeling procedure is that procedure is not always stable. There might be lots of procedure which have minor difference or parameter. In this case, the code would soon become complex and hard to be reused and even hard to be read and understand.

Take the following code snippet as example:

enterLoginDetailsForm.EnterLoginDetails(userName, password, appServer, dbName, companyCode);
defaultJurisdictionAndLocationSubLocationForm.DefaultJurisdictionAndLocationSubLocationTest(jurisdiction, location, timeZone);
everestTipoftheDayForm.EverestTipoftheDay_ClickBtnClose();

Pretty hard to figure out what exactly it do, right? Now check the following code for same procedure:

            // Detail comment would be added after confirmed EVTCM-764
            var loginDetail = new LoginDetailModel();
            loginDetail.Window.WaitForControlExist();
            loginDetail.UserCode.Text = UserName;
            loginDetail.Password.SendKeys(Password);
            loginDetail.OptionsExpandButton.Click();
            loginDetail.ApplicationServerName.Text = AppServer;
            loginDetail.DatabaseServerName.Text = DbName;
            loginDetail.Ok.Click();

            var jurisdictionWindow = new DefaultJurisdictionAndLocationSubLocationModel();
            jurisdictionWindow.Window.WaitForControlExist();
            jurisdictionWindow.SetJurisdiction(Jurisdiction);
            jurisdictionWindow.SetLocation(Location);
            jurisdictionWindow.SetTimeZone(TimeZone);
            jurisdictionWindow.Ok.Click();

            var tipWindow = new EverestTipoftheDayModel();
            if (tipWindow.Window.WaitForControlExist())
            {
                tipWindow.Close();
            }

It isn't this bit more clear than before? And there is also a good benefit that anywhere when you encounter the modeled dialog, you have it there ready to be reused.

An code example with Fluent API and Page-Object pattern can be found here

For the design pattern involved in the test automation, there is also another video talked this quite well 3 Design Patterns for More Reliable and Maintainable UI Test

Of course, the patterns are general guidelines. We don't need to match each of them exactly but we definitely should go towards them while we can.