How to use the Astade testrunner

The general idea of the testrunner is “stolen” from the “chat” program of the Linux ppp package.

Imagine your test target has a tcp/ip port to connect to. So a normal test would consist of sending commands and wait for the expected answers. For a telnet login that might be:

"ogin:" -> "thomas\n";
"assword:" -> "geheim\n";

This is the base of the testrunner syntax. In addition to this, the testrunner needs a timeout. Because, whenever the expected string does not arrive at all, after a while a timeout has to expire.

So this is a complete test script, which you could use to log in into a telnet server:

section login {
    timeout(10000) :
        "ogin:" -> "thomas\n";
        "assword:" -> "geheim\n";
};

To start the testrunner, copy the above testscript into a file named “test.txt” and call:

testrunner -h myhost.de -p 23 -s test.txt

The testrunner will:

1. connect to the host myhost.de:23
2. wait up to 10 seconds for the string “ogin:”
3. when receiving “ogin:” it will send “thomas\n”
4. wait up to another 10 seconds for the string “assword:”
5. when receiving “assword:” it will send “geheim\n”
6. than close the tcp connection

congratulations you did run you first simple test script.

The above test leaks of some very essential parts.

• First of all, we do have a timeout, but we did not specify, what should happen, if the timeout expires?
• Second, we do not get any test reporting, yet.

To solve this, we have to add some additional lines:

section login {
    watch: timeout -> report("timeout -> exit\n"), exit;
    timeout(10000) :
        always -> report("Test login begin\n");
        "ogin:" -> "thomas\n";
        "assword:" -> "geheim\n";
        always -> report("Test login successful!\n");  
};

This little test is pretty complete. It handles the timeout and it generates a nice test report (to std::cout).

The testrunner syntax consists of lists of “triggers” and “actions. So far we used 3 types of triggers:

1. TextTrigger, is a string (e.g.: “ogin:”)
2. AnyTrigger (always), triggers always
3. TimeoutTrigger (timeout), triggers when the timeout expires.

And we used 3 types of actions:

1. OutText (e.g.: “thomas\n”) sends a string to the tcp socket
2. ReportAction (e.g.: report(“Test login successful!\n)) sends a string to std::cout
3. exitAction (exit), ends the testrunner immediately

The language is recursive. That means, each section may contain sections! So we could extend our test, and use two subsections “login” and “ls”.

section myTest {
 
    watch: timeout -> report("timeout -> exit\n"), exit;
 
    section login {
        timeout(10000) :
            always -> report("Test myTest begin\n");
            "ogin:" -> "thomas\n";
            "assword:" -> "geheim\n";
    }
 
    section ls {
        timeout(2000) :
             "$" -> "ls\n";
             "$" -> none;
             always -> report("Test login successful!\n);  
    };
};

We have now two sub sections, with different timeouts. The trigger for the timeout is still in the top section. It is possible to do it this way. But we could have additional timeout triggers in the subsection, doing different things.

But be aware: if the timeout triggers inside a sub section, it will only trigger there. The event (timeout) is consumed and will not trigger in the main section!

In the section before, we have a main “section” and do some reporting, when it starts and when it ends. There is a special keyword for this behaviour: “test”. A test is a section, too. But it has additional functionality. One is, that the reporting at the begin and the end is already build in. So this would do exactly the same, as the example before:

test myTest {
 
    watch: timeout -> report("timeout -> exit\n"), exit;
 
    section login {
        timeout(10000) :
            "ogin:" -> "thomas\n";
            "assword:" -> "geheim\n";
    }
 
    section ls {
        timeout(2000) :
             "$" -> "ls\n";
             "$" -> none;
    };
};

If you want to generate test reports in xml format to import them in other tools (e.g. PTC) you can add a testnumber into the script, like this:

test login[34567] {
    watch: timeout -> report("timeout -> exit\n"), exit;
    timeout(10000) :
        "ogin:" -> "thomas\n";
        "assword:" -> "geheim\n";
};

if you run the testrunner, specifying an xml file:

testrunner -h localhost -p 40000 -s test.txt -x test.xml

you get a xml report:

<ActionList>
	<SetResult caseID="34567">
		<Annotation>
		</Annotation>
		<Verdict>Passed</Verdict>
	</SetResult>
</ActionList>

if you start the testrunner in verbose mode, you get texts in different colors.

• Your reports are in white (as always).
• In green you see, what is received from your test target.
• in yellow you see what your test script is sending to the test target.
• Fail and goto actions are displayed in red.

testrunner -v -h myhost.de -p 23 -s test.txt

• always : triggers always. It consumes all data from the input!
• shell(“ls -al”) : runs a shell command. If the return value of the shell command is non zero, all attached actions are executed. If it returns zero, the actions are not executed. It does not consume data.
• “text” : triggers, if a string compare is true. It consumes the data from the input including the trigger string. Data behind the trigger string is not consumed.
• timeout : triggers, if a timeout occurs. It consumes all data from the input!

• exit : the testrunner ends. This is for very hard errors only.
• fail : sets a flag in the running test, marking it as “failed”. At the end of the test there is a “fail” message written into the report, instead of a success message.
• goto(init) : this action leaves a section, starting another one. You can use this for simple conditional execution.
• none : does nothing.
• “test\n” : sends a text to the socket connection.
• report(“here we go!\n”) : is used to write additional information in the report (std::cout).
• shell(“ls”) : just executes a shell command.