The Position Tracking Performance Metrics(PTPM) Visualization and Analysis was created to help visualize and analyze data taken from a human tracking or detection program. In order to determine the effectiveness of that program it is generally necessary to simultaneously collect data from a system-under-test(SUT) and a ground-truth(GT) system. Differences between the two data sets are interpreted as errors in the system-under-test. Unless one uses simulated data the ground-truth is likely to also include errors. It is just assumed that they are negligible at-least for the purpose of analyzing a given data set.
The tool can be used on-line or off-line. Only the optitrack is currently supported to receive and display data on-line. Other sensors may be added in the future. For offline analysis and display the first step is to open a ground-truth(eg. gt.csv) and system-under-test(eg. sut.csv) file.
Install Netbeans 7.3 or higher and JDK 7+(optionally included with Netbeans) and ant 1.8.4 or higher (also included with Netbeans) from http://www.oracle.com/technetwork/java/javase/downloads/index.html and/or http://netbeans.org/.
From within Netbeans open and select the project "PositionTrackingPerformanceMetrics" by using the menu option "File->Open Project".
Build the project using the F11 key. (Or use "Run -> Build Project" menu option or use the icon with the hammer.)
Run the program from within Netbeans using the F6 key.(Or use "Run -> Run Project" menu option or use the icon with the green triangle pointing to the right.)
The ant program can compile the program on the command line. (However, it is not recommended. Conflicts with ant properties and environment variables often cause problems, such as not being able to find a jar file or using the wrong version of a compile tool)
Commands should be run from the top-level directory which contains the build.xml file.
To remove old temporary files:
ant clean
To build the output jar file:
ant jar
To build everything including javadoc (if configured) run ant with no arguments.
It is common for linux systems to have incompatible versions of ant on the path. In order to make sure the version of ant is the version which came with netbeans the environment variables PATH and ANT_HOME may need to be modified:
On most systems the program can be run with default options by double clicking the file PositionTrackingPerformanceMetrics.jar in the dist subdirectory.
Running from the command-line terminal allows the program to run on systems where options need to be modified, allows additional options to be specified, provides more feed-back on problems or potential problems and allows multiple sets of example files to be processed within a script.
The command line should be of the form:
<path-to-java> <java-options> -jar <path-to-PositionTrackingPerformanceMetrics.jar> <PTPM-options>
See http://docs.oracle.com/javase/7/docs/technotes/tools/windows/java.html for information on the java options.
The PTPM options include:
The panel on the left side contains a tree structure that allows the user to selected any ground-truth(GT) or system-under-test(SUT) track. A track is a series of position measurements and associated data of a single entity taken periodically for some time during the experiment. The main reason for selecting a track is to make it easier to identify in the panel on the right. The right panel an overhead view of the tracks. By default unselected ground-truth tracks are red, selected ground-truth tracks are orange, unselected system-under-test tracks are blue and selected system-under-test tracks are purple.The solid circle indicates the current location of each person the circle surrounding it shows the area a robot must avoid to ensure no contact with that person. The colored part of the track shows the path that person has followed so far (up to the current time selected with the slider at the bottom left). The grey part of the track shows the path that will be followed after the currently selected time.
The data in tracks can be read or written to CSV(comma-separated variable) files. These files are ASCII text files but are typically imported or exported from a spreadsheet program. Each line of the file provides a position at one instance of time for one object. Lines are separated by the new-line character. An optional carriage return character before or after the new-line will be ignored. Fields within a line are separated by the comma character. Blank lines and lines that do not begin with a digit or decimal-point are ignored. This allows comments and column headers to be stored in the file. The first four fields(time,name,x,y) are required although the order could be different. Default values can be assumed if later fields are missing.
The following is the default way that fields are expected for data entries:
time,name,x,y,z,velx,vely,velz,confidence
When opening a CSV file the user will be asked for the column numbers for each field so many CSV files that contain the same information but in different fields can still be read.
The default network protocol transfers data over a TCP socket using the same format as the CSV file. The PTPM GUI can act as either a client or as a server or both. The only difference between client and server modes is the order programs are started and the way the program it is connected to is programmed. To open the server port(s) check the boxes in the menu for "Connections->Open Port to Accept GT data" and/or "Connections->Open Port to Accept SUT data". One or more programs can then connect to the corresponding ports to stream data which will be displayed in (soft) real-time. Alternatively the program providing the data could be started first and written to accept connections on some port. The menu options "Connections->Connect to server of GT data" or "Connections->Connect to server of SUT data" would be used to complete the connection.
In addition to the data, a line starting with "source=" can be used to specify a label to make it easier to identify which sensor a track came from. (eg. "source=igps" )
The optitrack is a set of infrared strobe cameras used with special targets. TrackingTools software from NaturalPoint will stream positions of the detected targets using the NetNat protocol if so configured. The PTPM program can read data directly using this protocol. To do so use the "Connections->Connect/Show Live Optitrack Data" menu option.