vti(1)vti(1)NAMEvti - Visual Threads Instrumentation (VTI); detects potential race con‐
ditions and inconsistent lock ordering
SYNOPSISvti [-noinconsistent] [-nounguarded] [-customize filename]...
[-incobj lib]... [-excobj lib]... [-Lpath]... [-run|-display]
-prog {program} [argument...]
DESCRIPTION
VTI is an Atom tool that instruments your program to include analysis
checks for "unguarded" data which can result in race conditions, and
checks for deadlocks of multithreaded C and C++ programs at run-time.
You can use VTI for the following types of applications: For C or C++
applications using the Posix Threads Library that allocate memory by
using the malloc, calloc, realloc, valloc, alloca, sbrk, and new (C++
only) functions. (Applications that use either the -pthread or -threads
compilation option.) The application cannot use cancelability type of
PTHREAD_CANCEL_ASYNCHRONOUS. If your application uses a custom locking
package or memory allocator, you may still be able to use VTI by using
customization directives, or by using source code annotation direc‐
tives.
You can access VTI in two general ways: Invoking a vti command that
creates an instrumented version of your application in the current
working directory, and optionally runs the instrumented program and
displays the results of the analysis.
You may also choose to run the instrumented program at a later time
(independant of the vti command). In that case, you must set the
LD_LIBRARY_PATH environment variable to point to the directory(s) con‐
taining the instrumented executables before running the instrumented
program.
Whenever the instrumented program is run, the VTI tool reports race
conditions and inconsistent lock order violations in the log file with
the name program.vtilog. The graphical user interface of Visual
Threads (dxthreads) automatically invokes VTI when you choose Run from
the File menu, and you have enabled checking for certain rules
(UnguardedData or UnsafeFunction/UnsafeLibrary rules). Visual Threads
establishes the necessary environment for VTI, and after the instrumen‐
tation is complete, it invokes the instrumented image for you. Viola‐
tions are reported as popup Alarm boxes; additional information is pro‐
vided in the log file program.vtilog as well.
PARAMETERS
File name of a fully linked non-static executable to be analyzed. This
program should be compiled with the -g, -g2, or -g3 flag to obtain more
complete debugging information. If the default symbol table level
(-g0) is used, line number information, static procedure names, and
file names are unavailable. If -g1 is used, the variable name symbolic
information is not available.
Other restrictions: the program cannot be linked with the non_shared
option, the program cannot be compiled with the -pg option (profiled),
and the program cannot be stripped of symbols (strip).
You can specify a file name from another directory, but the instrumen‐
tation and log files created by VTI are created in (and optionally run
or displayed from), the current working directory. All arguments fol‐
lowing the program name are considered to be arguments needed to run
the instrumented program. (Arguments are ignored unless you specify
either the -run or the -display flag.) Multiple arguments can be spec‐
ified.
FLAGS
Specifies an additional filename to be used for customization direc‐
tives that control VTI instrumentation. By default, VTI checks for a
customization file in the current directory or in the home directory.
See "VTI Customization File". Multiple -customize flags can be speci‐
fied. VTI will not check for inconsistent lock ordering; this might be
useful if you only want to see reports on unguarded data race condi‐
tions. VTI will not check for unguarded data race conditions; useful
if you want to see only reports about inconsistent lock ordering.
Searches for shared libraries in the specified directory before search‐
ing the default directories. Use the same flags that were used when
linking the program with ld. Excludes the named shared library from
instrumentation. You can use the -excobj flag more than once to specify
several shared libraries. Note that certain libraries (such as
libc.so, libpthreads.so, and libpthread.so) are partially instrumented,
as needed by VTI, even if they are specified in the -excobj flag.
Instruments the named shared library. You can use the -incobj flag
more than once to specify several shared libraries. This option is
normally needed only for dynamic shared libraries (that is, libraries
loaded by calls to dlopen(3)), since vti instruments all non-dynamic
shared libraries by default. Execute the instrumented program, even if
no arguments are specified. By default, the program is just instru‐
mented for later execution. Execute the instrumented program, and run
more on the resulting .vtilog file(s).
NOTES
The algorithm for detecting "unguarded" data is as follows. For each
32-bit word that is not part of a thread stack, the tool allocates an
additional 32-bit word ('shadow memory'), which is used to keep track
of a candidate set of locks that might conceivably protect the memory
location. When a thread accesses the memory location, the location's
candidate set is intersected with the set of locks held by the thread.
If the candidate set becomes empty, the tool deduces that there is no
lock that protects the location, and reports an error.
This basic algorithm is augmented in a few ways to deal with memory
allocation, initialization, read-only data, read-write locks, error
suppression, and hardware atomic instructions, as follows: When memory
is newly allocated (for example by malloc()), the tool marks it as
"fresh"---not yet accessed by any thread. When a location is first
accessed, the tool does not keep track of the candidate set, but
instead remembers that the accessing thread "owns" the location, and
may access it without holding any locks. Maintenance of the candidate
set begins once the location is acessed by a second thread, at which
point the first thread loses ownership of the location. The tool keeps
track of whether a location has been written since it started maintain‐
ing the candidate set for the location. It will not report an error on
a location that has not been written. To accommodate read-write locks,
the tool keeps track of what locks are held in write (exclusive) mode,
and what locks are held in any mode. On read accesses, the intersec‐
tion of the candidate set is performed with the set of locks held in
any mode. On write accesses, the set of locks held in write (exclu‐
sive) mode is used. Once an error has been reported at a given loca‐
tion, further accesses of that location are ignored by the tool, until
the memory is reallocated (becomes fresh). The tool also keeps track
of all accesses to data via hardware atomic instructions (load-
locked/store_conditional instructions). If a subsequent access is made
to the same location without using a protected instruction, an error is
reported. This can be useful in detecting alignment-related race con‐
ditions. For example, consider a quadword structure which consists of
multiple fields, each accessed independently by multiple threads.
Depending on compiler alignment and exact usage by the program,
accesses to individual parts of the quadword may in fact be accesses to
the entire quadword; this can result in one thread inadvertently writ‐
ing the entire quadword instead of the intended field access.
More details of the algorithm can be found in the paper cited in
"Related Information" and in the sections of this description titled
"VTI False Alarms" and "VTI Test Coverage".
VTI also checks for potential deadlock conditions by checking for any
inconsistent use of lock ordering. For example, if mutex xm is first
locked before mutex ym , then any subsequent reversal of this lock
order will be reported. Inconsistent lock ordering is a common cause
of program deadlock.
The next sections describe the customization and annotation mechanisms
of VTI and the optional use of VTI environmental variables, which are
not necessary for the majority of users or applications. See the
"Examples" section for a straightforward commandline usage example.
VTI Customization File
VTI can take as input one or more user-specified customization file(s).
A customization file is used during instrumentation of your application
and allows you to refine and control the reporting of violations, and
to describe your own synchronization mechanisms and memory allocation
routines. A VTI customization file is generally not necessary if you
are using a Posix Threads Library interface (-pthread or -threads com‐
pilation option), and standard memory allocation routines from within a
C or C++ program.
VTI reads a system-wide .vti file for directives, and then looks for a
user-specific If the file is not in the current directory, VTI looks
for one in the home directory. If you are using VTI automatically from
Visual Threads and wish to use a customization file, you must ensure
that the customization file is created before invoking Visual Threads;
it should be in the working directory you specify to Visual Threads or
in the home directory.
The following general syntax rules apply to the .vti file: VTI treats
lines beginning with a pound-sign character (#) as comments. Procedure
names and file names may be specified within double-quotes. Procedure
names should match those printed by the nm command. In particular, C++
names must include the list of argument types, for example, han‐
dler(int). Inlined procedures must be compiled with the cxx -noinline
command (or similar mechanism) in order to be independantly instru‐
mented or referenced. Some .vti file directives include source loca‐
tion specifications with the following syntax:
[[library^]source_file^]procedure [line num]
Each of the components (library,source_file,procedure) may contain
shell-style pattern matching characters: * ? []
The library and source_file will be full matches if they contain
slashes, otherwise they will match only on the specified components of
the pathnames.
For example, the following directive disables instrumentation for race
conditions of all routines named foobar in the application: ignore foo‐
bar
The following directive disables instrumentation of all routines named
str in the library libc.so. ignore libc.so^*^str*
The following directive disables instrumentation at line 17 in the pro‐
cedure bar in the source file ../wombat.c ignore ../wombat.c^barline 17
Some .vti file directives include register specifications with the fol‐
lowing syntax: the procedure's (non-floating point) return value one of
the procedure's first six integer arguments an arbitrary integer regis‐
ter
Registers other than r0 are sampled at the start of the procedure. r0
(retval) is sampled at the end of the procedure. See below for exam‐
ples of the register specification.
Use the following memory-related options in the .vti file: Declare that
the specified procedure causes some memory to be freed or allocated.
The memory is set to the "just allocated" state, and locking informa‐
tion previously gathered for it is forgotten. Location must specify a
procedure; the optional line number specification is not allowed for
this directive. Register must be an integer register. Len is the
length in bytes of the memory.
For example, the following directive declares that procedure foo
returns a pointer to a 32-byte region that is freshly allocated: fresh‐
mem retval 32 foo
The following declares that procedure bar_free in library libbar.so
takes a pointer to a 64-byte block of memory that is freshly allocated:
freshmem a0 64 libbar.so^*^bar_free
This directive is equivalent to the source annotation vti_set (regis‐
ter, VTI_FRESH, 0, len) at the appropriate point in the source. Spec‐
ify file as another .vti file. If the filename starts with a slash
("/"), it is treated as a full pathname. Otherwise, it is interpreted
as a pathname relative to the directory containing the file that
includes it. Don't instrument for race conditions at the given loca‐
tion. Turn off error reporting for each thread as it enters the speci‐
fied procedure(s) and turn it on again when the thread exits the proce‐
dure. While the thread has error reporting turned off, no unguarded
data or inconsistent lock errors will be reported. In addition, no
lock orderings will be remembered. This is equivalent to the use of
the vti_ignore(1) source annotation at the start of the procedure and
vti_ignore(0) at the end.
Location must specify a procedure; the optional line number specifica‐
tion is not allowed for this directive. Results may be confusing if one
procedure specified in the ignoreall directive calls another procedure
for which ignoreall has been specified, because error reporting will be
turned on again by the exit from the inner procedure. Declare that
memory returned from, or passed to a given procedure is to be ignored
for the purposes of unguarded data checks. Violations will not be
reported on the memory unless its state is modified by a freshmem
directive, or a vti_set source annotation.
Location must specify a procedure; the optional line number specifica‐
tion is not allowed for this directive. The location is instrumented,
but any detected race conditions for that location will be suppressed
and will not appear in the log file.
Use the following lock-related options in the .vti file: Indicates that
procedure is a procedure that destroys the mutex or read-write lock
that is its first argument. procedure must be a simple procedure name.
This is equivalent to putting the source annotation
vti_destroy_lock(lock) at the beginning of the procedure. Indicates
that procedure is a procedure that acquires the mutex or read-write
lock that is its first argument. procedure must be a simple procedure
name. This is equivalent to putting the source annotation
vti_write_lock(lock) at the beginning of the procedure. Indicates that
procedure is a procedure that acquires the read lock that is its first
argument. procedure must be a simple procedure name. This is equiva‐
lent to putting the source annotation vti_read_lock(lock) at the begin‐
ning of the procedure. Indicates that procedure is a procedure that
releases the read lock that is its first argument. procedure must be a
simple procedure name. This is equivalent to putting the source anno‐
tation vti_read_unlock(lock) at the beginning of the procedure. Indi‐
cates that procedure is a procedure that attempts to acquire the mutex
or read-write lock that is its first argument, but without blocking.
It returns 0 on success, and another value on failure. procedure must
be a simple procedure name. This is equivalent to calling the source
annotation vti_write_lock(lock) in the procedure when the call suc‐
ceeds. Indicates that procedure is a procedure that attempts to
acquire the read lock that is its first argument, but without blocking.
It returns 0 on success, and another value on failure. procedure must
be a simple procedure name. This is equivalent to calling the source
annotation vti_read_lock(lock) in the procedure when the call succeeds.
Indicates that procedure is a procedure that releases the mutex or
read-write lock that is its first argument. procedure must be a simple
procedure name. This is equivalent to putting the source annotation
vti_write_unlock(lock) at the beginning of the procedure.
VTI Source Code Annotations
In some cases, it may be easier to directly annotate your source code
to teach VTI about your own synchronization mechanisms and/or memory
allocation routines. Annotations provide functionality similar to the
VTI customization file directives for memory and locking routines.
Some examples of source code annotations include: Tell VTI to ignore
errors on variable locations. Tell VTI that variable locations is
freshly allocated. Tell VTI that a write lock or mutex at address
mutex has been acquired. Tell VTI that a write lock or mutex at
address mutex has been released.
To use source code annotations, reference the header file
/usr/include/vti.user.h in your modules, and link your application with
the archive library /lib/libvti.a as in this example:
cc -g -o t t.c -pthread -lvti
The file /usr/include/vti.user.h has complete details about source code
annotations.
VTI Environment Variables
VTI also provides environment variables that control and provide more
information about VTI at runtime. The customization file(s) and source
directives can be used to control VTI behavior at instrumentation time,
while the environment variables are used when the instrumented applica‐
tion is run.
If you are using VTI automatically from Visual Threads, you must define
any VTI environment variables you want to use before invoking Visual
Threads.
The environment variable VTI_ARGS can have the following values: When
used in conjunction with the environment variable VTI_TRACEMUTEX,
report all acquisitions and release of traced locks, not just the first
acquisitions which define the lock order. Log all VTI messages to file
instead of the default program.vtilog. "-" means stdout and "--" means
stderr. Don't shadow the data segments of shared libraries. Don't
suppress duplicates of unguarded data violations by PC. Normally, if
there are multiple unguarded data violations for a given program PC,
VTI suppresses those duplicate reports. Don't suppress duplicates of
unguarded data violations, by data word. Normally, each data word in
the program can produce only one unguarded data violation report
(unless the data is freed and reallocated). The shadow heap is nor‐
mally allocated starting at address 0x28000000000. For most programs
this is a good choice; however, if your application uses memory in a
less-typical manner which happens to conflict with this region, you may
see an error message from VTI:
drd_shadow_allocate: no space.
In this case, you should specify a shadow heap start address that does
not conflict with your application's use of memory. Shadow writable
user mmaped regions, in order to do race detection on such regions.
This is not done by default. Log all regions shadowed, and the loading
of instrumented libraries.
Use the following environment variables for more information about
locking: Include a whitespace-separated list of hex addresses to trace
VTI state transitions. This is useful to get more information about an
unguarded data race condition at a particular address. For example:
setenv VTI_TRACE 0x140001bec Include a whitespace-separated list of
decimal mutex identifiers. This is used to get more information about
a deadlock involving multiple mutexes. For example:
setenv VTI_TRACEMUTEX "11 13"
You would typically determine the list of mutex identifiers by first
running the VTI tool and noting the inconsistent order violation and
its associated mutexes. Then you would define VTI_TRACE_MUTEX based on
those associated mutex identifiers, and rerun the VTI tool.
VTI Log File
VTI places its output in a log file that includes the a description of
each unguarded data and inconsistent lock usage violation. If the
application does a fork or vfork operation, VTI creates another log
file for analysis of the child process, including the child pid as part
of the log file name.
Note that if the application does a vfork, VTI replaces this operation
with a fork in the instrumented program. This is necessary because VTI
requires initialization of system and pthread libraries which does not
occur on a vfork operation.
VTI False Alarms
When testing for unguarded data, VTI assumes that the programmer has
followed a particular programming discipline: all shared writable vari‐
ables are protected by mutexes or read-write locks. The tool allows
for unlocked initialization of such variables as a special case.
In general, the tool does not understand synchronization mechanisms
other than locks and hardware atomic instructions. As a special case,
the tool generates no error messages when exclusive access to a vari‐
able is ensured only by the synchronization that is implicit in thread
creation/join operations. The tool does not understand more general
uses of thread creation/join synchronization, nor other synchronization
mechanisms, such as semaphores.
In addition, the tool does not automatically notice if the program
starts to protect a memory location with a different lock from one it
used previously, unless the memory location has passed through a known
memory allocator. For example, if a program maintains its own free
lists of memory regions, it may be able to reuse memory for program
data locations, but the tool is not aware that a particular location
has been reallocated for a new purpose. Therefore, the tool is likely
to give spurious error reports (false alarms) when applied to a program
that uses alternative synchronization mechanisms or application-spe‐
cific memory allocation routines.
It is usually possible to suppress these false alarms with a small num‐
ber of source code annotations or directives in the .vti file. Typi‐
cally, one would declare memory as "fresh" when passing it from one
thread to another, or when it becomes protected by a new lock, such as
when it is reallocated after spending time on a free list. This can be
done by using the vti_set annotation with the VTI_FRESH option, or by
using the freshmem directive. In some cases, it is best to have the
tool "ignore" such memory, by using the VTI_IGNORE option to a vti_set
source annotation, or by using the ignoremem directive.
VTI Test Coverage
The algorithms used by the VTI tool can detect potential race condi‐
tions and deadlocks that did not actually occur when the instrumented
program was run. Moreover, the algorithms are largely independent of
timing and scheduling variations: if the same threads execute the same
statements, the same locations will be reported as unguarded and the
same lock orderings will be reported as inconsistent, regardless of the
order in which the threads ran.
However, the timing of the program can alter when the errors are
reported, and where in the program they are first observed. As one
would expect, VTI may report different sets of errors on different runs
if those runs execute different parts of the code, either because the
computation itself is timing dependent, or because different input data
were used.
VTI cannot find errors in code that was not executed, so it is impor‐
tant to exercise all the parts of the instrumented program that you
wish to check. In addition, to find unguarded data, the memory must be
accessed by at least two threads or VTI will not know that the memory
is shared.
VTI cannot detect errors between threads of different address spaces
(for example, two processes using shared memory).
There is one circumstance in which VTI may miss unguarded data as a
result of variations in scheduling. This occurs when a thread allo‐
cates some memory, initializes it, makes it available to other threads,
but then accesses the memory again before any other thread has accessed
it, and without holding an appropriate lock. If this happens, the tool
will erroneously assume that the last access is part of the initializa‐
tion of the memory, and will not report an error. It is possible for
the user to eliminate this error by explicitly marking the end of the
initialization code with a source annotation that declares the memory
as being protected by the appropriate lock. For example:
vti_set (&variable, 0, &lock, sizeof (variable));
See /usr/include/vti.user.h for more details about source annotations.
EXAMPLES
Invokes vti to instrument program, producing the executable pro‐
gram.vti, running the executable program.vti with two arguments, and
displaying the output results from the log file program.vtilog.
Note that if you don't specify either of the -run or the -display
flags, you must ensure that the environment variable LD_LIBRARY_PATH
points to the instrumented program and instrumented libraries before
running the instrumented program. You may also need to define the
environment variable PTHREAD_RESTRICTED_DISTRIBUTION, on some versions
of the operating system.
Typical output from the program.vtilog file might be something like:
data race by thread 3 on stl t0, 0(v0) accessing 0x140000300 at: rand
test1.c:109 (test1:0x120001ccc test1.vti:0x12003f394) thread_fun
test1.c:121 (test1:0x120001d40 test1.vti:0x12003f4e8) thdBase
?file?:? (libpthread.so:0x3ff805812e4
libpthread.so.test1.vti:0x3ff808420c4)
location was previously accessed readonly by multiple threads thread 3
holds locks: <none>
This log file output indicates that there was an unguarded data viola‐
tion (race condition) at source line number 109 in the function rand,
source file test1.c. (The function rand was called by function
thread_fun, which was created by the thdBase function in the pthreads
library.) The program location that was "unguarded" was at address
0x140000300. This location was previously accessed by multiple
threads, but the current thread doesn't hold any locks and is modifying
the program location.
FILES
Instrumented version of program. Log of race conditions, deadlocks,
and other reports. The name of this log file can be overriden by the
settings of the VTI_ARGS environment variable. Customization file for
VTI. Header file for compilation of an application which includes VTI
source annotations. Archive library to link with an application which
includes VTI source annotations.
RELATED INFORMATIONatom(1)
Programmer's Guide
"Eraser: A dynamic race detector for multithreaded programs", by S.Sav‐
age, M.Burrows, C.G.Nelson, P.G.Sobalvarro, T.E.Anderson in ACM Trans‐
actions of Computer Systems, Vol 15, No. 4, Nov 1997, pp. 391-412.
dxthreads(1)pthread(3)vti(1)
