Robot tests – Testing Plone

Robot tests

Robot tests#

Robot tests are written in test suites, which are plain text files, usually with the filename ending .robot.

Robot test file format#

We can see an example of robot tests in our package. plonecli creates an example robot test file in tests/robot/test_example.robot:

*** Settings *****************************************************************

Resource  plone/app/robotframework/selenium.robot
Resource  plone/app/robotframework/keywords.robot

Library  Remote  ${PLONE_URL}/RobotRemote

Test Setup  Open test browser
Test Teardown  Close all browsers


*** Test Cases ***************************************************************

Scenario: As a member I want to be able to log into the website
  [Documentation]  Example of a BDD-style (Behavior-driven development) test.
  Given a login form
   When I enter valid credentials
   Then I am logged in


*** Keywords *****************************************************************

# --- Given ------------------------------------------------------------------

a login form
  Go To  ${PLONE_URL}/login_form
  Wait until page contains  Login Name
  Wait until page contains  Password


# --- WHEN -------------------------------------------------------------------

I enter valid credentials
  Input Text  __ac_name  admin
  Input Text  __ac_password  secret
  Click Button  Log in


# --- THEN -------------------------------------------------------------------

I am logged in
  Wait until page contains  You are now logged in
  Page should contain  You are now logged in

A robot test file is composed of different sections:

  • Settings: where we can import test libraries, resource files and variable files. We can define metadata for test suites and test cases and we can also define some actions to perform when a test starts (Setup) or finishes (TearDown), such as opening the browser or closing it.

  • Variables (not used in this example): a section where we can define some variables that could be used everywhere in the test.

  • Test Cases: a list of test scenarios based on available keywords.

  • Keywords: a set of actions that we can perform during the test.

Basically, a robot test is all about running test clauses (keywords). Every test case may contain one or more keywords, which are run sequentially – usually until one fails.

Keywords can have arguments, and are separated from their arguments (and arguments from each other) using at least two spaces.

Note

Keywords are defined in keyword libraries and as user keywords, and they can be imported into a test in the Settings section.

Keyword libraries can be Python libraries or XML-RPC services. User keywords are just lists of test clauses reusing existing keywords or other user keywords. User keywords are described in the test suite, or imported from resource files.

Test scenarios#

There are several ways to write robot test scenarios. In Plone we choose the Given-When-Then style.

This format is very useful because allows you to write test cases that everyone can understand, as they are written in a familiar way.

When you write test cases in this style, you express the initial state with a keyword starting with word Given, actions with keywords starting with the word When and the expectations with a keyword starting with the word Then.

Note

Keyword starting with And or But may be used if a step has more than one action.

Let's see how a keyword works. In our example, we have this scenario:

Scenario: As a member I want to be able to log into the website
  [Documentation]  Example of a BDD-style (Behavior-driven development) test.
  Given a login form
   When I enter valid credentials

The initial state is the keyword that starts with "Given":

  Given a login form

This keyword is a set of sequential actions (defined in the Keywords section):

a login form
  Go To  ${PLONE_URL}/login_form
  Wait until page contains  Login Name

First, we open the login_form page. We wait for the page to be loaded and for the two fields to be available in HTML.

Note

${PLONE_URL} is a global variable defined in an plone.app.robotframework imported library.

After a Given statement, there is a list of actions (When), and a final expectation (Then) that has the same structure.

Note

Most of these actions come from the default Selenium library (imported in the Settings section). You can find a list of available actions online.

For standard Plone actions and keywords, see the imported files (keywords.robot and selenium.robot).

Running robot tests#

A package created with plonecli is already configured to run robotframework tests.

To use robotframework tests in your package, you need the plone.app.robotframework dependency in the setup.py file.

There is also a robot part in the base.cfg file. This configuration isn't required to run robot tests, but it installs two helper scripts for writing tests:

  • bin/robot-server starts a temporary Plone site with the given test layer set up

  • bin/robot executes Robot Framework’s pybot-runner so that it will run the given test suite against the running robot-server, ensuring that tests will be run in isolation (i.e., the database is cleaned up between tests)

[robot]
recipe = zc.recipe.egg
eggs =
    ${test:eggs}

We can run robot tests along with other tests, using this command:

plonecli test --all

Or, as with any other test, we can run only a single test using this command:

plonecli test -s plonetraining.testing -t test_example.robot --all

These commands take time because each test case needs to start a server, open a new browser window and then execute keywords and await the server reponse.

When we are developing our tests, we can speedup this process, keeping a robot-server instance always up with this command:

bin/robot-server --reload-path src plonetraining.testing.testing.PLONETRAINING_TESTING_ACCEPTANCE_TESTING

This command will start a robotframework server that also instantiates a Plone instance.

This has a second advantage: you can inspect the Plone site every time and try things manually before writing actions in the test.

With robot server running, we can run test cases with this command:

bin/robot /src/plonetraining/testing/tests/robot/test_example.robot

Note

These development helpers worked well, up to and including Plone 5.1. At the moment, there are problems starting a robot-server instance using Plone 5.2 and WSGI.

Debugging robot tests#

Debugging a robot framework test can be hard because there could be problems in the server or in the client.

Test execution could be very slow and difficult to follow for a human, so we can slow it down to better understand what's happening:

*** Settings ***

Suite setup  Set Selenium speed  2s

Alternatively, we could pause (or set a sleep timeout on) the execution of a test, and use the time to manually inspect the site:

*** Test Cases ***

Pause tests with included Pause-keyword
  Pause

Pause tests for 10 minutes and then continue
  Sleep  10 min

Finally, we could also pause the test execution and test keywords using the console:

*** Test Cases ***

Start interactive debugger
    Import library  DebugLibrary
    Debug

Note

There are more detailed examples in the Plone robot framework documentation.

Test reports#

If we run a robot framework test using the plonecli test command, we have the usual console output that tells us if a test succeeded or failed.

Robot tests are a combination of backend and frontend environments, so it isn't easy to have all information about the test status in the console.

For that reason, the robot framework generates detailed reports in the parts/test folder where you can see the status of the last test execution and detailed information on each test case.

Exercise 1#

Let's write our first robot test!

Try to write some basic scenarios:

  • Login to the site and create a new TestType content item

  • Visit the TestType view, passing a custom message in the query string

Note

plonecli created a basic robot test for our TestType content type, so the first part of the exercise could be copied from it. Try not copying it and try using the robot framework syntax by yourself.

In the robot framework documentation you can find some information that can help you write your first robot framework test.

Solution
*** Test Cases ***************************************************************

Scenario: As a site administrator I can add a TestingItem
  Given a logged-in manager
    and an add TestingItem form
   When I type 'My TestingItem' into the title field
    and I submit the form
   Then a TestingItem with the title 'My TestingItem' has been created

Scenario: As a site administrator I can view a TestingItem
  Given a logged-in manager
    and a TestingItem 'Foo'
   When I go to the TestingItem view
   Then I can see the TestingItem title 'Foo'

Scenario: I can pass a custom message and see it in the page
  Given a logged-in manager
    and a TestingItem 'Foo'
   When I go to the testing-item-view view with a custom message 'Hello trainers'
   Then I can see my message 'Hello trainers' in the page

*** Keywords *****************************************************************

# --- Given ------------------------------------------------------------------

a logged-in manager
  Enable autologin as  Manager

an add TestingItem form
  Go To  ${PLONE_URL}/++add++TestingItem

a TestingItem '${id}'
  Create content  type=TestingItem  id=${id}  title=${id}

# --- WHEN -------------------------------------------------------------------

I type '${title}' into the title field
  Input Text  name=form.widgets.IBasic.title  ${title}

I submit the form
  Click Button  Save

I go to the TestingItem view
  Go To  ${PLONE_URL}/foo
  Wait until page contains  Site Map

I go to the testing-item-view view with a custom message '${message}'
  Go To  ${PLONE_URL}/foo/testing-item-view?message=${message}
  Wait until page contains  Site Map


# --- THEN -------------------------------------------------------------------

a TestingItem with the title '${title}' has been created
  Wait until page contains  Site Map
  Page should contain  ${title}
  Page should contain  Item created

I can see the TestingItem title '${title}'
  Wait until page contains  Site Map
  Page should contain  ${title}

I can see my message '${message}' in the page
  Wait until page contains  Site Map
  Page should contain  ${message}

Exercise 2#

Now let's test something that we can't test using functional tests: JavaScript.

  • Add a basic mockup pattern to testing-item-view (for example, the autotoc pattern)

  • Check that the table of contents is rendered on the page

Solution

First, add autotoc to the template:

<html xmlns="http://www.w3.org/1999/xhtml"
      xmlns:metal="http://xml.zope.org/namespaces/metal"
      xmlns:tal="http://xml.zope.org/namespaces/tal"
      xmlns:i18n="http://xml.zope.org/namespaces/i18n"
      i18n:domain="plonetraining.testing"
      metal:use-macro="context/main_template/macros/master">
<body>
  <metal:block fill-slot="content-core">
      <h2 i18n:translate="">Sample View</h2>
      <p>This is the default message: ${view/msg}</p>
      <p tal:define="custom_msg view/custom_msg"
         tal:condition="custom_msg">This is the custom message: ${view/custom_msg}</p>
      <div class="pat-autotoc"
         data-pat-autotoc="scrollDuration:slow;levels:h4,h5,h6;">
        <h4>Title 1</h4>
        <p>Mr. Zuckerkorn, you've been warned about touching. You said
            spanking. It walked on my pillow! How about a turtle? I've always
            loved those leathery little snappy faces.</p>
        <h5>Title 1.1</h5>
        <p>Ah coodle doodle do Caw ca caw, caw ca caw. Butterscotch!</p>
        <h6>Title 1.1.1</h6>
        <p>Want a lick? Okay, Lindsay, are you forgetting that I was
            a professional twice over - an analyst and a therapist.</p>
        <h4>Title 2</h4>
        <p>You boys know how to shovel coal? Don't worry, these young
        beauties have been nowhere near the bananas. I thought the two of
        us could talk man-on-man.</p>
      </div>
  </metal:block>
</body>
</html>

Now, create a test case that checks autotoc:

*** Test Cases ***************************************************************

Scenario: I can see toc in testing-item-view
  Given a logged-in manager
    and a TestingItem 'Foo'
   When I go to the testing-item-view
   Then I can see the table of contents

*** Keywords *****************************************************************

# --- Given ------------------------------------------------------------------

a logged-in manager
  Enable autologin as  Manager

an add TestingItem form
  Go To  ${PLONE_URL}/++add++TestingItem

a TestingItem '${id}'
  Create content  type=TestingItem  id=${id}  title=${id}

# --- WHEN -------------------------------------------------------------------

I go to the testing-item-view
  Go To  ${PLONE_URL}/foo/testing-item-view
  Wait until page contains  Site Map
# --- THEN -------------------------------------------------------------------

I can see the table of contents
  Wait until page contains  Site Map
  Page Should Contain Element  css=.autotoc-nav

Exercise 3#

For the last JavaScript test, try to simulate clicks.

  • The autotoc pattern can also be used for generate tabs.

  • Check that clicking on tabs results in changes to the document object model (DOM).

Solution

Add tabs to the template:

<html xmlns="http://www.w3.org/1999/xhtml"
      xmlns:metal="http://xml.zope.org/namespaces/metal"
      xmlns:tal="http://xml.zope.org/namespaces/tal"
      xmlns:i18n="http://xml.zope.org/namespaces/i18n"
      i18n:domain="plonetraining.testing"
      metal:use-macro="context/main_template/macros/master">
<body>
  <metal:block fill-slot="content-core">
      <h2 i18n:translate="">Sample View</h2>
      <p>This is the default message: ${view/msg}</p>
      <p tal:define="custom_msg view/custom_msg"
         tal:condition="custom_msg">This is the custom message: ${view/custom_msg}</p>
      <div class="pat-autotoc"
         data-pat-autotoc="scrollDuration:slow;levels:h4,h5,h6;">
        <h4>Title 1</h4>
        <p>Mr. Zuckerkorn, you've been warned about touching. You said
            spanking. It walked on my pillow! How about a turtle? I've always
            loved those leathery little snappy faces.</p>
        <h5>Title 1.1</h5>
        <p>Ah coodle doodle do Caw ca caw, caw ca caw. Butterscotch!</p>
        <h6>Title 1.1.1</h6>
        <p>Want a lick? Okay, Lindsay, are you forgetting that I was
            a professional twice over - an analyst and a therapist.</p>
        <h4>Title 2</h4>
        <p>You boys know how to shovel coal? Don't worry, these young
        beauties have been nowhere near the bananas. I thought the two of
        us could talk man-on-man.</p>
      </div>
      <div class="pat-autotoc autotabs"
         data-pat-autotoc="section:fieldset;levels:legend;">
        <fieldset>
          <legend>Tab 1</legend>
          <div>Foo</div>
        </fieldset>
        <fieldset>
          <legend>Tab 2</legend>
          <div>Bar</div>
        </fieldset>
        <fieldset>
          <legend>Tab 3</legend>
          <div>Baz</div>
        </fieldset>
      </div>
  </metal:block>
</body>
</html>

Update the previous test with more test cases:

# ============================================================================
# DEXTERITY ROBOT TESTS
# ============================================================================
#
# Run this robot test stand-alone:
#
#  $ bin/test -s plonetraining.testing -t test_autotoc.robot --all
#
# Run this robot test with robot server (which is faster):
#
# 1) Start robot server:
#
# $ bin/robot-server --reload-path src plonetraining.testing.testing.PLONETRAINING_TESTING_ACCEPTANCE_TESTING
#
# 2) Run robot tests:
#
# $ bin/robot /src/plonetraining/testing/tests/robot/test_autotoc.robot
#
# See the http://docs.plone.org for further details (search for robot
# framework).
#
# ============================================================================

*** Settings *****************************************************************

Resource  plone/app/robotframework/selenium.robot
Resource  plone/app/robotframework/keywords.robot

Library  Remote  ${PLONE_URL}/RobotRemote

Test Setup  Open test browser
Test Teardown  Close all browsers


*** Test Cases ***************************************************************

Scenario: I can see toc in testing-item-view
  Given a logged-in manager
    and a TestingItem 'Foo'
   When I go to the testing-item-view
   Then I can see the table of contents

Scenario: I see Tab 1 value by default in testing-item-view
  Given a logged-in manager
    and a TestingItem 'Foo'
   When I go to the testing-item-view
   Then I can see the content of tab ('Foo') and not 'Bar'

Scenario: I see Tab " value when i select it in testing-item-view
  Given a logged-in manager
    and a TestingItem 'Foo'
   When I go to the testing-item-view
    and I click 'Tab 2'
   Then I can see the content of tab ('Bar') and not 'Foo'


*** Keywords *****************************************************************

# --- Given ------------------------------------------------------------------

a logged-in manager
  Enable autologin as  Manager

an add TestingItem form
  Go To  ${PLONE_URL}/++add++TestingItem

a TestingItem '${id}'
  Create content  type=TestingItem  id=${id}  title=${id}

# --- WHEN -------------------------------------------------------------------

I go to the testing-item-view
  Go To  ${PLONE_URL}/foo/testing-item-view
  Wait until page contains  Site Map

I click '${tab}'
  Click Link  xpath=//a[contains(text(), "${tab}")]

# --- THEN -------------------------------------------------------------------

I can see the table of contents
  Wait until page contains  Site Map
  Page Should Contain Element  css=.autotoc-nav

I can see the content of tab ('${visible}') and not '${invisible}'
  Wait until page contains  Site Map
  Element Should Be Visible  xpath: //div[contains(text(), "${visible}")]
  Element Should Not Be Visible  xpath: //div[contains(text(), "${invisible}")]