xtorx(1) - gui for the torx on-the-fly tester
Table of Contents
xtorx - gui for the torx on-the-fly tester
xtorx [ options ... ]
configuration-file ...
xtorx(1) reads the configuration file(s),
if given, and opens a window that contains the following from top to bottom:
a Menu bar, a Button bar, an Executed test steps pane, a Spec pane, a
Verdict bar and a Messages pane. The window manager may add a Window Title,
usually at the top.
Many buttons and menu entries are enabled or disabled
depending on the state in which xtorx(1) is. For example, buttons and menu
entries that set test execution parameters are only enabled while no test
execution is taking place, i.e. either when xtorx(1) has just been started,
or after a test execution has come to an end. A test exection can come to
an end in two ways: 1) when an error is found, the test execution run ends
automatically, and 2) the user can end a running test execution by clicking
the Stop button in the Button bar.
The Window Title displays information
about the current configuration, primers, and mutants files, and about
currently selected primer and mutant. This same information is also added
to the Messages pane at the start and end of each test run.
The Menu bar
contains a File menu to deal with files (and to clean the Messages pane,
and to exit), a Preferences menu, a Primers and a Mutants menu to choose
a primer or a mutant (see below), a View menu to view a primer or a mutant
source file (when available and/or configured), a Tools menu to enable
or disable visualiation tools for the test run, and a Help menu. The Primers,
Mutants, View, and Tools menus are context dependend; their menu-entries
depend on the configuration file loaded and the primers and mutants files
(again, see below) that are loaded.
The Button bar contains the buttons
to start and stop a test execution run, and to switch between manual (single-stepping)
and automatic test execution mode. When a test execution has been started,
xtorx(1) is by default in manual mode. In manual mode, the user is in full
control, and test steps can be done using the buttons from the Spec pane.
In automatic mode, xtorx(1) controls the testing, by repeatedly executing
test steps, making its own decisions about observing and stimulating. The
automatic mode can be enabled by clicking the Auto radio-button in the Mode
field of the Button bar, after which automatic testing will immediately
begin! Clicking the Manual button in the Mode field switches back to manual
mode. If a positive integer value has been entered in the Steps box in the
Button bar when the automatic mode is enabled, xtorx(1) will try to execute
the specified number of test steps, after which it will switch back to
manual mode.
The Executed test steps pane contains the trace (steps executed
so far) of the current execution run. Each test step that is executed is
appended here, preceded by the test step number.
The Spec pane contains
the test actions for the current state (possible inputs, or stimuli, and
expected outputs, or observations), together with buttons to stimulate
(Random Input, Selected Input) to observe (Output), or to do an arbitrary
test step, where xtorx decides between stimulating and observing (Random).
Double-clicking on an input action will execute this action. Additionally,
if a trace of a previous test execution run is replayed, the `current' action
of the trace will be shown, together with a button (Use Trace) to re-execute
this action.
The Verdict bar displays the status of testing process. If
the pane has color red it means failure. If the color is green it means
test purpose was hit or trace is passed (when an attempt was done to replay
a log as a trace). If the color is orange it means test purpose was missed
or inconclusive (was unable to completely replay the given trace).
The
Messages pane is mostly used for logging and debugging purposes. It will
contain the output produced by the SUT (System Under Test), if any, together
with debugging messages of various TorX tool components. The messages are
grouped together for each test execution run by lines (one above and one
below all messages for the test run) that start with a triangular button,
followed by the configuration parameters for that test run. Depending on
the version of the wish (tcl/tk) interpreter that is being used (xtorx
is a tcl/tk script), it is possible to hide and show the messages for each
individual test execution run by clicking the first (usual left) mouse
button on the triangular button. Clicking the second (usual middle) mouse
button on a triangular button scrolls the window to make its ``partner'' visible.
The main window of xtorx(1) is divided vertically in three main
panes. The top pane consists of the Title bar, the Button bar and the Executed
test steps pane. The middle pane contains the Spec pane, and the Verdict
bar. The bottom pane justs consists of the Messages pane.
The relative size
of these panes can be changed by clicking and dragging the first (usally
left) mouse button in the small square resize buttons that appear on top
of the pane borders. How the panes are resized can be influenced with the
Preferences -> OpaquePaneResize setting.
The window title displays
the current configuration, primers and mutants file(s), and the selected
primer and mutant (if any). The window title is added by the window manager;
not all window managers do add one, though.
The menu bar contain
all required operations for testing. The operations are divided into the
categories File, Preferences, Primers, Mutants, View, Tools, and Help. We
will discuss each of them in its own section.
The File menu contains
all operations which involve operations with files.
- Open Configuration...
- Open
a configuration file. If the Init Gui From Config and Reset Gui From Config
toggle buttons in the Preferences menu are set, a number of settings and
menu entries will be initialised from the information in the configuration
file. The following items will initialised (the corresponding torx-config(4)
keywords appear between parentheses): log file (LOGFILE), trace file (TRACEFILE),
seed (SEED), primers file and Primers menu (PRIMERS), selected primer (PRIMER),
guides file and Guides menu (GUIDES), selected mutant (GUIDE), mutants
file and Mutants menu (MUTANTS), selected mutant (MUTANT), View menu (SPECSOURCE
and IUTSOURCE), and visualization Tools menu (LOGMON).
- Open Trace...
- Choose
a trace file that is to be used in the next test execution run.
- Close Trace
- Do not use a (the previously) chosen trace file in the next test execution
run.
- Open Primers...
- Select a primer configuration file, and if this is succesful,
update the Primers menu, and enable the Primers menu button and its menu
entries.
- Close Primers
- Clear the Primers menu, and disable the Primers menu
button, and deselect the primer, if one was chosen from the Primers menu.
- Reopen Primers
- Reloads the primer configuration file, to refresh the Primers
menu when the primer configuration file is changed.
- Open Mutants...
- Select
a mutant configuration file, and if this is succesful, update the Mutants
menu, and enable the Mutants menu button and its menu entries.
- Close Mutants
- Clear the Mutants menu, and disable the Mutants menu button, and deselect
the mutant, if one was chosen from the Mutants menu.
- Reopen Mutants
- Reloads
the mutant configuration file, to refresh the Mutants menu when the mutant
configuration file is changed.
- Set Logfile Name...
- Select the logfile name
that is to be used in the next test execution run.
- Unset Logfile Name...
- Clears
the logfile name, i.e. do not create a log in the next test execution run.
- Clear Messages
- Clear the Messages pane.
- Save Messages to File...
- Save the messages
in the Messages pane to the specified file.
- Exit
- Exit the program. This will,
however, not exit ``independent'' visualization tools, even if they are started
from xtorx(1).
In the Preferences menu the following options
can be enabled or disabled.
- Start MSC
- If this option is set when the Start
button is pressed, the mscviewer(1) will be started via xtorx-showmsc(1)
to visualize the test run. (default value: enabled)
- Set MSC Window Reuse
Button
- This option sets the initial value of the Reuse button of the mscviewer(1).
(default value: enabled)
- Use Bigger Fonts
- Change all fonts to a different
one (that should be bigger). This option is meant for demo purposes. (default
value: disabled)
- Init Gui From Config
- If this option is set, settings indicated
in the description of the Open Configuration... command from the File menu
will be updated if corresponding entries are present in the configuration
file that is opened. (default value: enabled)
- Reset Gui From Config
- If this
option is set, settings indicated in the description of the Open Configuration...
command from the File menu will be reset, unless the Init Gui From Config
option is enabled, and the configuration file contains corresponding entries.
(default value: enabled)
- OpaquePaneResize
- An unpronounceable option with
means that the contents of the panes are moved as well during resizing
of panes (while the mouse button is pressed; otherwise, only the pane separator
lines are moved, and actual resizing of the pane contents is only done
once the mouse button is released). (default value: disabled)
- Enable Trace
Support
- Add a Use Trace button and a trace text field to the Spec pane,
and an AutoTrace button next to the Auto button in the Button bar (by default,
these are hidden to save screen space). It may be necessary to resize the
xtorx(1) window, to make it wider, to make these buttons visible. (default
value: disabled)
- Enable Instantiation Support
- Add a Use Instantiation button
and an instantiation text entry field to the Spec pane (by default, these
are hidden to save screen space). It may be necessary to resize the xtorx(1)
window, to make it wider, to make these buttons visible. (default value:
disabled)
- Show Message Buttons
- Add a Clear Messages button and a Save Messages
to File... button to the end of the Messages pane (by default, these buttons
are hidden to save screen space). (default value: disabled)
- Show Refresh
Buttons
- Add a Refresh button and a auto- toggle button (by default: enabled)
to the end of the Button bar. If the auto- toggle button is disabled, the
Spec pane (in particular: the lists of possible inputs and expected outputs)
will not automatically be updated after a test step, but only after the
Refresh button has been pressed (by default, these buttons are hidden to
save screen space). (default value: disabled)
The Primers menu
gives a list of primers of which one can be selected. This list is generated
from the primers file that is loaded via the File -> Open Primers menu entry
(or automatically via a PRIMERS configuration file entry, as described
above). If a PRIMER entry is present in the configuration file, it will
be used to set the default. Otherwise, if an entry none is present in the
Primers menu, it will be the default. Otherwise, initially none of the primers
will be selected.
Depending on the details of a particular configuration,
the test execution configuration parameters may be incomplete when no primer
is selected, which may cause test execution runs to fail even before a
test step has been done. So, if a test execution run fails in this way,
make sure to check if you have forgotten to select a primer.
A side effect
of selecting a primer in the menu can be that additional, primer-specific,
configuration file(s) are loaded, like for example the .torx files generated
by mkprimer(1). Such a file may contain, for example, a primer-specific SPECSOURCE
entry.
Of course, what exactely happens when a primer is selected depends
completely on the contents of the primers file that was loaded. For a description
of what can be specified in a primers file, see xtorx-extension(n). Examples
of primers files can be found in the torx-examples distribution.
The
Guides menu gives a list of guides of which one can be selected. This list
is generated from the guides file that is loaded via the File -> Open Guides
menu entry (or automatically via a GUIDES configuration file entry, as
described above). If a GUIDE entry is present in the configuration file,
it will be used to set the default. Otherwise, if an entry none is present
in the Guides menu, it will be the default. Otherwise, initially none of
the guides will be selected.
Depending on the details of a particular configuration,
the test execution configuration parameters may be incomplete when no guide
is selected, which may cause test execution runs to fail even before a
test step has been done. So, if a test execution run fails in this way,
make sure to check if you have forgotten to select a guide.
A side effect
of selecting a guide in the menu can be that additional, guide-specific,
configuration file(s) are loaded, like for example the .torx files generated
by mkprimer(1). Such a file may contain, for example, a guide-specific GUIDESOURCE
entry.
Of course, what exactely happens when a guide is selected depends
completely on the contents of the guides file that was loaded. For a description
of what can be specified in a guides file, see xtorx-extension(n). Examples
of guides files can be found in the torx-examples distribution.
The
Mutants menu gives a list of mutants of which one can be selected. This
list is generated from the mutants file that is loaded via the File -> Open
Mutants menu entry (or automatically via a MUTANTS configuration file entry,
as described above). If a MUTANT entry is present in the configuration file,
it will be used to set the default. Otherwise, if an entry none is present
in the Mutants menu, it will be the default. Otherwise, initially none of
the mutants will be selected.
Depending on the details of a particular configuration,
the test execution configuration parameters may be incomplete when no mutant
is selected, which may cause test execution runs to fail even before a
test step has been done. So, if a test execution run fails in this way,
make sure to check if you have forgotten to select a mutant.
A side effect
of selecting a mutant in the menu can be that additional, mutant-specific,
configuration file(s) are loaded, like for example the .torx files generated
by mkprimer(1). Such a file may contain, for example, a primer-specific IUTSOURCE
entry.
Of course, what exactely happens when a mutant is selected depends
completely on the contents of the mutants file that was loaded. For a description
of what can be specified in a mutants file, see xtorx-extension(n). Examples
of mutants files can be found in the torx-examples distribution.
The
View menu contains two entries to view the source of the specification
resp. the implementation (or mutant).
- Primer source
- show the source file
of the primer using xtorx-showspec(1). This button is only enabled if the
torx-config(4) configuration file contained an SPECSOURCE (or, deprecated,
SOURCESPEC) entry.
As mentioned above in the section about the Primers menu, it is possible
to set up a primers file in such a way that selecting a primer from the
menu causes an additional primer-specific configuration file to be loaded,
that defines (o.a.) a SPECSOURCE entry for the selected primer.
- Mutant source
- show the source file of the mutant using xtorx-showspec(1). This button is
only enabled if the torx-config(4) configuration file contained an IUTSOURCE
(or, deprecated, SOURCEIUT) entry for the selected primer.
As mentioned above in the section about the Mutants menu, it is possible
to set up a mutants file in such a way that selecting a mutant from the
menu causes an additional mutant-specific configuration file to be loaded,
that defines (o.a.) a IUTSOURCE entry for the selected mutant.
The
Tools menu contains a list of toggle buttons to enable and disable visualization
tools, or more generally, tools that work on the torx-log(4) log file of
a test execution run. This list is generated from LOGMON entries in the
torx-config(4) configuration file(s) when a configuration file is loaded
(or from a primer- or mutant-specific configuration file when a primer or
mutant is selected). By default, all entries in the list are enabled.
The Help menu contains the following entries to get more information
and browse (query) and submit problem reports.
- Help on TorX
- (to be implemented)
- About TorX
- displays a dialog box containing copyright and contact information.
- On Version
- (to be implemented)
- Query Problem Reports (using tkgnats)...
- Invokes
torx-querypr(1) to open a tkgnats(1) window to query problem reports.
- Query
Problem Reports (using web-browser)...
- (to be implemented)
- Report Problem (using
tkgnats)...
- Invokes torx-sendpr(1) to open a tkgnats(1) window to submit problem
reports. This can be used to report problems about the tool, inconsistencies,
etc. to us.
- Report Problem (using web-browser)...
- (to be implemented)
The Button bar contains buttons and text entry fields to control the
execution of a test run:
- Start button
- Start a test execution run. This means
that the torx(1) program is started under the control of xtorx(1); xtorx(1)
is merely a graphical wrapper around torx(1).
- Stop button
- Stop the test
execution run, by asking the torx(1) program that was started via the Start
button to quit (exit).
- Seed field
- here the seed can be entered for the random
number generator that torx(1) and the other TorX components will use. If
the user has not filled in this entry when the Start button is pressed,
xtorx(1) will itself randomly choose a value and fill in the field. By default,
this field is empty. Once filled, xtorx(1) will not overwrite it.
- Manual
mode button
- swiches to the manual mode of on-the-fly testing. In this mode
the user is in complete control, and can use the buttons in the Spec pane.
This is the default mode, that is entered every time that the Start button
is pressed,
- Auto mode button
- switches to automatic mode of on-the-fly testing.
In this mode xtorx(1) will make all decisions; it is like a user who continously
presses the Random button in the Spec pane. If the Steps field is filled
with an integer value when the Auto button is pressed, only the specified
number of steps will be done, after which xtorx(1) will switch back to
manual mode. When an error is found while running in Auto mode, the test
run is ended. Warning: when this mode is selected, xtorx(1) will immediately
start (continue) running the test.
- AutoTrace mode button
- This mode is only
available when a trace file is loaded, and this button is only visible
when Preferences -> Enable Trace Support is selected.
switches to the automatic trace mode of on-the-fly testing. This mode is like
a user who continously presses the Use Trace button in the Spec pane. If
the Steps field is filled with an integer value when the AutoTrace button
is pressed, only the specified number of steps will be done, after which
xtorx(1) will switch back to manual mode. When an error is found while running
in AutoTrace mode, the test run is ended. Warning: when this mode is selected,
xtorx(1) will immediately start (continue) running the test.
- Steps field
- this field can be used to specify the number of test steps that should
be done (at most) when the Auto or AutoTrace button is pressed (if an error
is found before the specified number of test steps is done, the test will
be ended).
- auto- toggle button
- This button is only visible if the Preferences->
Show Refresh Buttons setting is enabled.
When the auto- toggle button is enabled, the lists of inputs and outputs
events in the Spec pane are automatically updated after each test step.
Otherwise, these lists are only updated after the Refresh button is pressed.
(default value: enabled)
- Refresh button
- This button is only visible if
the Preferences-> Show Refresh Buttons setting is enabled.
Update the lists of inputs and outputs events in the Spec pane.
The Executed test steps pane displays a trace of the test
steps which have been executed: inputs that have been sent as simulus to,
or outputs that have been received as observation from the SUT. Each test
step is preceded by the test step number. The test step that is currently
being visualized (or highlighted) by the tools that were enabled in the
Tools menu when the test run was started has a yellow background. It is
possible to change the `currently visualized' test step in xtorx by clicking
the third mouse button in a test step, or dragging the mouse over the test
steps with the third button down. Note that also each of the individual
visualization tools can be used to change the `currently visualized' test
step.
The Spec pane has two lists next to each other, each in its
own sub-pane: a list of Inputs and a list of Outputs. During a test execution
run, the Inputs list contains the possible input events (possible stimuli
that can be sent to the SUT) for the current state, and the Outputs list
contains the expected output events (observations that are expected from
the SUT). If a test execution run is ended because an error is found, the
Inputs list will be empty, and the Outputs list will contain the expected
observations. At the same time, the last event in the Executed test steps
pane is the last (erroneous, conflicting, invalid) event that was received
from the SUT.
The relative horizontal sizes of these sub-panes can be changed
by clicking and dragging the first (usally left) mouse button in the small
square resize buttons that appears on top of the pane border. How the panes
are resized can be influenced with the Preferences -> OpaquePaneResize setting.
Under the Inputs and Outputs lists the following buttons are present to
control individual test steps during the test execution run:
- Selected Input
- send the input event that is selected in the Inputs list as stimulus to
the SUT.
- Random Input
- let the program randomly select an input event and
send it to the SUT.
- Random
- let the program randomly decide between stimulating
and observing, and then, depending on the result of this ``decision'', behave
as if the Random Input resp. the Output button was pressed.
- Output
- get an
observation from the SUT, and check if it is in the list of expected output
events.
If instantiation support was enabled (see the Preferences -> Enable
Instantiation Support button above), under the buttons mentioned above
another button and a text entry field are added. When an event is selected
in the Inputs list, it is copied to the text entry field, where it can
be edited. The copied event may contain a predicate (enclosed between square
brackets ``['' and ``]'') that consists of one or more constraints on the values
of the variables in the event. The constraints are separated by ``;'', there
may be an optional ``;'' after the last constraint, before the closing ``]''. When
editing the event, the predicate may be deleted; the constraints can be
used as inspiration when choosing values for the variables. Note that it
is not mandatory to choose values for all variables -- it is also possible
to change the constraints (or add new ones) to reduce the number of possible
values for the variables, and let ``the system'' (an instantiator(1)?) then
come up with a single value. The button can be used to try to apply the
edited event, in a similar way as the other buttons in the Spec pane:
- Use
Instantiation
- try to apply the action shown in the instantation field as
the next input event, i.e. check if the instantiation is a valid one, and,
if so, use it as a stimulus and send it to the SUT.
If trace support was
enabled (see the Preferences -> Enable Trace Support button above), under
the buttons mentioned above another button and a text field are added. The
text field is used to display the subsequent event from the trace, and
the button can be used to apply it, in a similar way as the other buttons
in the Spec pane:
- Use Trace
- try to apply the action shown in the trace
field as the next input or output event, i.e. if the action in the trace
field is an input event, use it as a stimulus and send it to the SUT, and
otherwise, if the action in the trace field is an output event, get an
observation from the SUT and check if it is valid (in the list of output
events), and if it is identical to the action shown in the trace field.
This bar contains a text field in which the verdict will be
given at the end of a test execution run. Depending on the particular verdict,
the color if the bar will be changed.
The pane is colored red when an error was found (usually this means that
an observation received from the SUT was not in the list of expected output
events) (in tradional terms: fail).
The pane is colored green when a test purpose was hit or, when an attempt
was done to replay a log as a trace, the end of the trace was succesfully
reached without finding an error (in tradional terms: pass).
The pane is colored orange when a test purpose was missed or, when an
attempt was done to replay a log as a trace, the actual test run deviated
from the trace, but without finding an error (in tradional terms: inconclusive).
The Messages pane is mostly used for logging and debugging
purposes. It will contain the output produced by the SUT (System Under Test),
if any, together with debugging messages of various TorX tool components.
The messages are grouped together for each test execution run by lines
(one above and one below all messages for the test run) that start with
a triangular button, followed by the configuration parameters for that
test run. Depending on the version of the wish (tcl/tk) interpreter that
is being used (xtorx is a tcl/tk script), it is possible to hide and show
the messages for each individual test execution run by clicking the first
(usual left) mouse button on the triangular button. Clicking the second
(usual middle) mouse button on a triangular button scrolls the window to
make its ``partner'' visible.
Currently, it seems, there are no X
defaults setting to be used for configuration.
- *.if
- torx(1) torx-config(4)
configuration files
- *.primers
- primers files
- *.mutants
- mutants files
torx-intro(1), torx(1), torx-config(4), torx-log(4), xtorx-extension(n), xtorx-showspec(1),
xtorx-showmsc(1), anifsm(1), aniwait(1), jararacy(1), mscviewer(1), torx-querypr(1),
torx-sendpr(1)
It is not possible to instantiate variables in events
by hand when giving the Selected Input command.
Sometimes, during a test
execution run, xtorx(1) may get into a state in which all ``useful'' buttons
are disabled. In such a case, the only sensible thing to do is to use the
File -> Exit button to exit, and use the ps(1) and kill(1) command to check
for, and kill, runaway TorX processes.
Occasionally it happens that when
xtorx is started, in the window only the Messages pane is visible. The solution
is to use the small square resize button, half of which is hidden under
the menu bar, to resize the panes. When the Spec pane is visible again,
it is very well possible that there only the Outputs are visible (the Inputs
are hidden as well). The solution is here as well to use the small square
resize button that is now completely at the left of the window, to resize
the Outputs pane to make the Inputs visible.
A number of commands in the
Help menu have remained unimplemented too long.
By Email: <torx_support@cs.utwente.nl>
This manual page documents version 3.9.0 of xtorx.
Table of Contents
|
Prev
|
Table of Contents
|
Next
|
|
Appendix D: TorX Manual Pages: xtorx-showspec(1) - show a specification file
|
|
Appendix D: TorX Manual Pages: torx-config(4) - configuration file for torx tester
|