Privacy & Legal Notice
Table of Contents
- Part 1: The Very Basics
- Overview
- Supported Platforms and Languages
- Starting TotalView
- Environment Setup
- Compiling Your Program
- Starting TotalView
- TotalView's Basic Look and Feel
- Primary Windows
- Dialog Boxes
- Mouse Usage
- Menus
- Accelerator Keys
- Scrolling and Resizing
- Process and Thread State Codes
- X Resources
- TotalView's Basic Functions
- Viewing Source and Assembler Code
- Setting a Breakpoint
- Controlling Execution
- Diving Into Objects
- Viewing and Modifying Data
- Text Editing and Searching
- Saving Window Contents
- Getting Help
- Exiting TotalView
- Part 2: Additional TotalView Functions
- Loading Executables
- Attaching / Detaching Processes
- Viewing a Core File
- Expression Evaluation
and Code Fragments
- More on Action Points
- Setting Executable Command
Arguments
- Setting Source Code Search
Paths
- Setting stdin, stdout, and stderr
- Signal Handling
- Setting Preferences
- Visualizing Array Data
- Command Line Interpreter (CLI)
- Part 3: Debugging Parallel Codes
- Process/Thread Groups
- Debugging Threaded Codes
- Overview
- Compiling Threaded Codes
- Finding Thread Information
- Selecting a Thread
- Execution Control for
Threaded Programs
- Viewing and Modifying Thread
Data
- Debugging MPI Codes
- Overview
- Compiling MPI Codes
- Starting an MPI Debug Session
- Selecting an MPI Process
- Controlling MPI Process
Execution
- Viewing and Modifying
Multi-process Data
- Displaying Message Queue
State
- Debugging OpenMP Codes
- Overview
- Compiling OpenMP Codes
- Debugging OpenMP Programs
- PRIVATE, SHARED and THREADPRIVATE
Variables
- Debugging Hybrid Codes
- Overview
- Compiling Hybrid Codes
- Debugging Hybrid Programs
- Batch System Debugging
- Miscellaneous
- Topics Not Covered
- References and More Information
- Appendix A: TotalView Intrinsic
Variables and Built-In Statements
- Workshop Home
|
Overview
|
|
|
What is TotalView?
- TotalView is a sophisticated software debugger product of
Etnus LLC.
- Used for debugging, analyzing, and tuning program performance.
- Especially designed for use with complex, multi-process and/or
multi-threaded applications.
- Has been selected as the Department of Energy's
Accelerated Strategic Computing
Initiative (ASCI) debugger.
Key Features of TotalView:
- Provides source and assembler level debugging for serial, parallel,
multi-process and multi-threaded codes.
- Portable: able to be used in a variety of UNIX environments, including
those with distributed, clustered, uniprocessor and SMP machines.
- Supports most popular parallel programming models/libraries such as MPI,
OpenMP, Threads, PVM, HPF, Shmem, fork/exec and hybrid.
- Provides all debugging facilities through easy to learn and use
X-Windows based Graphical User Interface. Also provides a command
line interpreter for non-GUI debugging.
- Can be used to debug a specified program, an unattached running process,
or a core file.
- On a per process/thread basis, permits you to view:
- Source code, assembler code, or both interleaved
- Source for called functions
- The execution stack trace (procedure calling stack)
- Stack variables and registers
- Program data (variables, arrays)
- MPI message queues
- Provides for the insertion and execution of "code fragments" within the
current process context.
- Provides several types of "action points", as well as the ability to
set, delete, suppress, unsurpress and save them:
- process breakpoint - on a source line basis
- multi-process barrier - blocking breakpoint for parallel processes
- conditional breakpoint - where breakpoint occurs only if a code
fragment expression is satisfied
- evaluation points - where code fragments are evaluated
- Allows you to easily modify program data (addresses, arrays, array slices,
variables) while debugging
- Provides graphical visualization of array data during debugging session
- Includes an extensive hypertext based online help system
| Most features are supported on all platforms (next section). However,
due to differences in operating systems, not all features are supported
identically on
all platforms. For example, certain aspects of threads support,
watchpoints, and message queues support may vary across platforms.
|
How to Obtain TotalView:
- TotalView software and documentation (and more) are available from Etnus's
web site: www.etnus.com
- Software can be downloaded for free for a trial period. All
documentation may be downloaded for free anytime.
|
Supported Platforms and Languages
|
|
|
Etnus Supported Platforms:
Supported Languages:
- C
- C++
- Templates
- Inlined functions and code in header files
- Base and virtual classes
- Fortran77
- Fortran90
- Fortran90 types
- Modules
- Assumed shape arrays
- Mixed C/C++ and Fortran
- Assembler
Non-Etnus Platforms:
Ports of TotalView to other platforms are also available, or under
development from several other vendors. Note: These implementations may
differ from the Etnus product and will not be covered by this tutorial.
Detailed Platform Information:
|
Starting TotalView
|
|
|
Environment Setup
Before you can start using TotalView, you will most likely need to
specify several environment related parameters. Some of these are
site dependent and may already be setup for you.
Path Variable:
Depends upon where the TotalView product is installed
on your system. For example, if TotalView was installed on an AIX
platform in a directory called
/usr/local/packages/TotalView , then
you might do something like:
set path=($path /usr/local/packages/TotalView/rs6000/bin)
License File:
TotalView is typically installed under the FLEXlm
server. This location will vary from site to site. For example:
| csh/tcsh
| setenv LM_LICENSE_FILE /usr/local/flexlm/license.dat |
| bsh/ksh
| export LM_LICENSE_FILE=/usr/local/flexlm/license.dat |
DISPLAY Variable and xhost Permissions:
Because the TotalView GUI is X-Windows based, you will probably need to
set the usual X-Windows display related parameters. For example, if your
terminal/screen is called
xterm4.local.net, and the machine
where you are actually going to run TotalView to debug a program is called
node12.lab.gov, then you would
do both of the following:
1. On xterm4.local.net
you would issue the command:
xhost node12.lab.gov
2. On node12.lab.gov you
would issue the command:
| csh/tcsh
| setenv DISPLAY xterm4.local.net:0 |
| bsh/ksh
| export DISPLAY=xterm4.local.net:0 |
UNIX Trusted Hosts:
For multi-process programs that run on multiple machines (like MPI),
you will probably need to insure that you've established the usual
UNIX trusted hosts mechanisms for all of the machines you plan to
use. Minimally, you'll need to be able to make sure that
rsh (or a substitute command) works.
Compiling Your Program
-g:
- Like many UNIX debuggers, you will need to compile your program with the
appropriate flag to enable generation of symbolic debug information.
For most compilers, the -g option is used for this.
- TotalView will allow you to debug executables which were not compiled with
the -g option. However, only the assembler code can be
viewed.
Beyond -g:
- Don't compile your program with optimization flags
while you are debugging it. Compiler optimizations can "rewrite"
your program and produce machine code that doesn't necessarily match
your source code.
- Parallel programs: usually require you to use a special compiler
or specific compiler flags. Some examples are discussed later in the
Debugging
Parallel Codes section.
- Some compilers (such as PGI HPF) may require additional compilation flags.
See the TotalView Users Guide
for details.
Starting TotalView
A Variety of Ways:
TotalView can be started in a number of different ways, depending upon
whether you want to:
- debug an executable file
- attach to a running process
- debug a core file
- change the debugger's default appearance/behavior
Some Examples:
| Command / Action
|
- totalview
- Starts the debugger. You can then load a program or
corefile, or else attach to a running process.
- totalview filename
- Starts the debugger and loads the program specified by
filename.
- totalview filename corefile
- Starts the debugger and loads the program specified by
filename and its core file specified by corefile.
- totalview filename -a args
- Starts the debugger and passes all subsequent arguments
(specified by args) to the program specified by filename. The
-a option must appear after all other TotalView options on
the command line.
- totalview filename -remote hostname[:portnumber]
- Starts the debugger on the local host and the totalview debugger server
on the remote host hostname. Loads the program specified by
filename for remote debugging. You can specify a host name or
TCP/IP address for hostname, and optionally, a TCP/IP port number
for portnumber.
- totalview -bg blue -fg white
- Starts the debugger with a blue background color and a white
foreground (font) color.
- totalview -fn 9x15
- Starts the debugger and uses specified font. Useful if
the default font is too small.
|
|
|
|
|
|
|
More Information:
- The totalview command has quite a number of options
(over 80) that may be used to control the GUI's behavior and appearance.
- Additionally, it is possible to customize TotalView's appearance and
behavior through "initialization files". These initialization files
can be global to a site and/or local to a user.
- See the
TotalView Users Guide
for more information about command options and initialization files.
|
TotalView's Basic Look and Feel
|
|
|
Primary Windows
Root Window:
- Will always appear when the TotalView GUI is started. If TotalView is
started without any arguments, it is the only window to appear.
- Comprised of:
- 5 Pull-down menus - File, Edit, View, Tools, Window, Help
(menus are discussed later)
- 4 "Pages" - Attached, Unattached, Groups, Log.
- Attached Page - displayed by default (shown above).
For each program or process that is being debugged, it shows:
- Machine where it is running
- Process ID
- Status code
- Name of program or process
- Number of associated threads (if any)
- Small triangle toggle. If threads are present, clicking this toggle
will show each associated thread with its thread ID, status
and current routine of execution.
-
Unattached Page - shows the list of processes (if any) that
you own
but are not currently debugging. These processes can be "attached to"
for debugging. Discussed later.
-
Groups Page - Lists the process/thread groups associated
with your program. Discussed later.
-
Log Page- Displays a log of TotalView debug information.
Process Window:
- Usually (but not always) appears with the Root Window after
TotalView is started.
- By default, a single process window will display. For multi-process /
multi-threaded programs however, every process and every
thread may have its own Process Window if desired.
- Comprised of:
- 10 Pull-down menus
- 10 Execution control buttons
- 4 Navigation control buttons
- Process and thread status bars
- 5 "Panes"
- Stack Trace Pane
- Shows the call stack of routines the current executable is running
- May select any routine from the call stack - will update the Source
Pane and Stack Frame Pane automatically.
- Source Pane
- Displays source/assembler for the currently selected program or
function.
- Shows program counter, line numbers and any associated action points.
- Only "boxed" line numbers are eligible for debugging.
- Stack Frame Pane
- Displays the local variables, registers and function parameters for
the selected executable.
- Register abbreviations and meanings are architecture specific. See
Appendix C: Architectures in the
TotalView Users Guide for
details.
- Thread List Pane
- Shows all threads that exists within the process.
- Selecting any thread from the list automatically updates the Source,
Stack Frame and Stack Trace panes.
- Action Points Pane
- Lists all breakpoints, barrier points, evaluation points and watchpoints
for the process.
- Provides a convenient way to quickly select and modify action points
which may be scattered widely over a program.
Variable Window:
- Appears when you dive on a variable or select a menu item to view
variable information.
- Comprised of a single pane and 5 pull-down menus.
- Displays detailed information about selected program variables. Also
permits editing of variable data.
Dialog Boxes
- TotalView has numerous dialog boxes that are used for a variety
of purposes:
- Solicit and confirm selections
- Display informational, warning and error messages
- Accept input
- Display and select options and preferences
- Display various types of information
- A few representative dialog boxes are shown below.
Mouse Usage
Much of your interaction with the TotalView debugger is through the use
of a three-button mouse. Each mouse button has a specific purpose,
described below.
NOTE: If you are using a Windows or Mac machine that doesn't have a
three button mouse, you will probably need to configure the mouse you
have to emulate a three button mouse.
| Mouse Button
| Purpose
| Description
| LEFT
| Select / Dive
| Single clicking on an object causes it to be selected and/or to
perform its action. Double clicking allows you to dive into an
object. For example, clicking on an array object in the source
pane will cause a new window to pop open, showing the array's values.
|
| MIDDLE
| Paste
| Writes information previously copied or cut into the clipboard
at the cursor's position. Not supported in all windows.
|
| RIGHT
| Context menu
| Pops-up a menu of common commands related to the object clicked on.
Supported in most windows and panes, but not dialog boxes.
| |
Menus
Two Types of Menus:
- Pull-down menus
- Appear at the top of most windows
- Activated by clicking with the left mouse button
- Some menu selections may have submenus
- Pop-up menus
- Activated by clicking on an object (such as a variable, line number,
etc.) with the right mouse button
- Not all objects possess pop-up menus
- Menus are context sensitive - different windows will have different
menus. Different versions of TotalView may display different menus.
- Menu selections that are dimmed are either irrelevant or not
available.
Root Window Menus:
Process Window Menus:
Variable Window Menus:
Pop-up Menu Examples:
Accelerator Keys
Short Cut:
- In addition to selecting actions from menus, you can also use
TotalView's predefined accelerator keys to initiate most of the
debugger's common functions.
- The action associated with an accelerator key depends upon which
TotalView window (Root, Process, etc.) has focus / is raised.
- You can always find out which accelerator key(s) to use by viewing the
menu for the action - accelerator keys are shown on the right side of
the menu.
Examples:
Scrolling and Resizing
Conventional Scrolling Behavior:
- Conventional scrollbars are used by most of TotalView's windows, pages
and panes.
- Scrolling can be accomplished by clicking and/or dragging with
with the left mouse button.
- The usual up-arrow, down-arrow, page up and page down keys can also
be used for scrolling.
Resizing Windows and Panes:
- All windows can be resized in the usual X-windows fashion by dragging
window borders with the mouse to a new size/position.
- The Process Window panes can be also be resized by clicking and
dragging on any resize widget.
Process and Thread State Codes
- TotalView uses colored, single character State Codes to describe process and
thread state information. For example, these codes are displayed in the
Process Window's Threads Pane:
- State codes also appear in the Root Window's
Attached and
Unattached pages.
- The table below summarizes TotalView's state codes.
| State Code
| Description
|
| B
| Stopped at a breakpoint
|
| E
| Stopped because of an error
|
| H
| In a Hold state
|
| I
| Idle
|
| K
| Thread is executing within the kernel
|
| M
| Mixed - some threads in a process are running and some not
|
| R
| Running
|
| S
| Sleeping
|
| T
| Thread is stopped
|
| W
| At a watchpoint
|
| Z
| Process in "zombie" state (child process that has
terminated and waiting on parent process to cleanup)
|
X Resources
- A number of TotalView's attributes and behaviors can be customized in the
usual X-Windows manner by setting them in your .Xdefaults file.
- The TotalView Users Guide
lists the X resources that can
be specified. Some examples of things that can be set:
- Background and foreground colors
- Window sizes and placements
- Action point behavior
- Source code search paths
- Signal handling
- Font size/type
- Visualizer properties and behavior
- PVM and HPF enablement, behaviors
- ...
- Starting with TotalView version 5, users can also specify some attributes
and behaviors through the Preferences Dialog Box (discussed later).
- As mentioned previously, the totalview command has over 80
options that may be used to control the GUI's behavior and appearance.
These will override the .Xdefaults settings for that TotalView session only.
|
| Due to the redesign of TotalView's GUI in version 5, X resources that
were used in previous versions of TotalView may or may not be valid.
Furthermore, some X resources have been "deprecated" - that is, they
are still supported, but may not be in future releases.
|
|
TotalView's Basic Functions
|
|
|
Viewing Source and Assembler Code
Source, Assembler or Interleaved:
Displaying Function / File Source Code:
Setting a Breakpoint
What Is a Breakpoint?
- A breakpoint is the most basic of TotalView's action points used to
control a program's execution. It causes a process/thread to halt execution
at the line number - prior to executing that line number.
- Breakpoints can be set in source code and assembler code.
- For regular source, only "boxed" line numbers are eligible for breakpoints.
For assembler or interleaved source, only assembler instructions displaying
a "gridget" are eligible.
- TotalView has three other types of action points (discussed later):
- Process barrier point
- Evaluation point
- Data watchpoint
Several Ways to Set / Unset a Breakpoint:
- Method 1: The easiest way to set a breakpoint is to
simply click on a
source code line number with the left mouse button. A red STOP icon
will then appear on the source line number, as shown below.
To unset the breakpoint, simply click on the red STOP icon.
It will then be deleted.
- Method 2: Right mouse click on the desired line number until
the pop-up menu appears (below). Then select Set Breakpoint.
- Method 3: First, click on a source line to select it (make sure
it's highlighted). Then use:
Process Window >
Action
Point Menu > Set Breakpoint
Selecting the Delete command from the Action Point Menu will remove
the breakpoint.
- Method 4: For any arbitrary line number:
Process Window >
Action
Point Menu > At Location
A dialog box will then open to prompt you for the line number.
Viewing Breakpoints:
- TotalView displays breakpoint information in several locations:
- As a "STOP" icon on the selected source line number
- Within the Action Points Pane
- Within the Action Point Properties Dialog Box
- In the Process Window's status bars
- In the Root Window state code column
Breakpoint Options:
- TotalView provides a means for selecting how breakpoints behave
across multi-process / multi-threaded programs. This topic is
further discussed in Part 3: Debugging Parallel
Codes.
Controlling Execution
- Controlling the execution of a program within TotalView involves two
decisions:
- Selecting the appropriate command
- Deciding upon the scope of the chosen command
- Both of these are performed via the
Process
Window, and are discussed below.
Execution Control Commands:
- TotalView enables you to control program execution in several different
ways. The table below describes the basic execution control commands.
| Command
| Description
|
| Go
| Start/resume execution
|
| Halt
| Stop execution
|
| Next
| Run to next source line or instruction. If the next
line/instruction
calls a function, the entire function will be executed and control will
return to the next source line or instruction (the function is
"stepped over").
|
| Step
| Run to next source line or instruction. If the next
line/instruction
calls a function, the function will be "stepped into". Execution will
stop within the function.
|
| Out
| Execute to the completion of a function. Returns to
the instruction after the one which called the function.
|
| Run To
| Allows you to arbitrarily click on any source line and
then run to that point.
|
| Next Instruction
| Similar to Next, but applies only to machine instructions
|
| Step Instruction
| Similar to Step, but applies only to machine instructions
|
| Hold/Release
| Hold ignores other commands to resume execution.
Release allows other run commands to have effect.
|
| Restart
| Restarts a running program, or one that has stopped without exiting.
|
| Set PC
| Sets the Program Counter to a desired source line, machine instruction,
or absolute address.
|
Group, Process, Thread Command Scopes:
- Most of TotalView's execution control commands can be applied at
the Group, Process or Thread scoping level. The two exceptions
are Restart (Group only) and Set PC (Thread only).
- Group scope:
- Executes the command on all processes within a specified group.
- Usually used for a multi-process parallel program.
- Can be used for a single process serial program (number of group members
= one).
-
Process Window >
Group Menu
- Process scope:
- For a multi-process program, executes the command on a single
selected process.
- Some subtleties in behavior exist if the process is multi-threaded,
but generally the command influences all threads owned by the process.
- Can be used for a single process serial program.
-
Process Window >
Process Menu
- Thread scope:
- Usually used to execute the command on a single thread of a
multi-threaded process.
- Behavior can differ between machine architectures. Not
supported on all platforms.
- Can be used for single process, single threaded serial program also.
-
Process Window >
Thread Menu
- Additional details about Group, Process and Thread commands usage are
discussed later in
Part 3: Debugging
Parallel Codes.
Diving Into Objects
- TotalView enables you to view more detail about a data containing
object (such as an array variable) by "diving" into it.
- Diving can be accomplished by three different methods:
- Double left clicking on an object
- Right clicking on an object and then selecting Dive from
the resulting pop-up menu (if applicable)
- Selecting Dive from any window's View menu
(if applicable)
- What happens when you dive on an object depends upon the object.
The table below describes most cases.
| Object
| Where Object is Located
| What Happens
|
| Process or thread
| Root Window
| Process/thread is displayed in an existing Process Window. If none
exists, then a new Process Window appears for the selected
process/thread.
|
| Routine
| Process Window (in Stack Trace Pane)
| Stack Frame and Source Code panes in the Process Window
are updated with information for the selected routine.
|
| Subroutine
| Process Window (in Source Code Pane)
| Source code appears in the Process Window
|
| Pointer
| Process Window
| Referenced memory area appears in a new Variable Window
|
| Variable, array, address
| Process Window
| Variable contents appear in a new a Variable Window
|
| Element of an array or structure
| Variable Window
| Contents of element appear in the Variable Window.
Example of a "nested" dive.
|
| Source code line
| Process Window
| Pop-up menu appears
|
Example:
Nested Dives and Undiving:
Viewing and Modifying Data
Viewing Data:
- TotalView allows you to view local variables, global variables,
registers, areas of memory and machine instructions, as discussed below.
- Local Variables
- Method 1: Dive on any variable that appears in the Source Pane or
Stack Frame Pane of the
Process Window.
- Method 2: Use the variable lookup command:
Process Window >
View
Menu > Lookup Variable
- Global Variables
- Registers
- Dive on any register that appears in the Stack Frame Pane of the
Process Window.
- Memory Areas
- Use the variable lookup command:
Process Window >
View
Menu > Lookup Variable
- Then enter either a hexadecimal address (must start with 0x) for a
single location. Enter two hexadecimal addresses for a range.
- Machine Instructions
- Method 1: Dive into the address of an assembler instruction in
the
Process Window Source Pane. The instructions for the entire
function will display in a Variable Window.
- Method 2: Dive on the PC in the Process Window Stack Frame Pane.
The instructions for the entire
function will display in a Variable Window.
- Leaving a Variable Window open allows you to perform runtime monitoring
of variables. TotalView will update its contents each time the program
is stopped.
Examples:
Modifying Variable Data:
- You can edit variables from within the Variable Window. Simply click on
the variable with the Select (left) mouse button. This will select
the variable for field editing.
- The Variable Window below demonstrates editing an array element.
Notice that the array element being edited (4,1) is enclosed in a box
with the field editor cursor.
- The modified variable has effect when the program resumes execution.
Arrays:
- For array data, TotalView provides several additional features:
- Displaying array slices
- Data filtering
- Data Sorting
- Array statistics
- Displaying Array Slices
- Used to display subsections of an array. Particularly useful if only a
small section of a large array is of interest.
- Can be entered in the Slice: field in the Variable Window.
- Syntax is lower_bound:upper_bound:stride and may be
specified for each dimension.
- Examples:
| Fortran
| Slice: (1:5, 3:8)
|
| C/C++
| Slice: [::2][1:20]
|
- Array Data Filtering
- Arrays containing data types of character, integer or floating
point can be filtered to display only desired data.
- Can be entered in the Filter: field in the Variable Window.
- Filtering can be:
- By arithmetic comparison
- For IEEE values
- By a range of values
- Within an expression
- Examples:
| Fortran
| Filter: .gt. 250
Filter: .eq. $nan
Filter: 7:.le.512
|
| C/C++
| Filter: >= 100
Filter: != $inf
Filter: 128:<1024
|
- Sorting Array Data
- Select either:
Variable Window >
View
Menu > Sort > Ascending
Variable Window >
View
Menu > Sort > Descending
- The Variable Window will be updated with the array elements sorted as
specified.
- Note: Sorting takes place internal to TotalView and not actually
within your data.
- Array Statistics
-
Variable Window >
Tools
Menu > Statistics
- A window containing statistical information about your array will
appear - example below.
- See the TotalView Users Guide
for additional examples, syntax options and other important information.
Changing Variable Display Format:
Changing Variable Data Types:
Text Editing and Searching
Text Editing:
- TotalView provides a basic field editor for use within certain debugger
fields and windows. Text which can be edited will be highlighted
and enclosed in an edit box with a field editor cursor. For example:
- Cutting and pasting can be accomplished by using the middle mouse
button or by selecting Cut, Copy, or Paste
from any window's
Edit pull-down menu.
Text Searching:
- Most TotalView windows will permit you to search for text strings.
- Simply select Find from any window's
Edit pull-down menu.
- A dialog box will appear for you to enter the string to search for,
plus any search options, as shown below:
- Select Find Again from the same
Edit menu to repeat a search.
Saving Window Contents
Getting Help
- Beginning with version 5, TotalView provides an extensive, hypertext
based online Help system.
- All primary TotalView windows have a Help pull-down menu
that includes links to Etnus's complete set of product documentation.
- Additionally, many dialog boxes have a context-sensitive Help
button.
- The Help pull-down menu and the hypertext Users Guide are
shown below:
Exiting TotalView
- You can exit the debugger in several ways:
-
Root
Window >
File
Menu > Exit
- Typing CTRL-q or CTRL-Q in any window
- Closing the Root Window via your X-windows window manager
- After selecting any of these ways to exit TotalView, you will be prompted
to confirm your choice to exit:
This concludes TotalView Part 1
What would you like to do?
Original Process Window (before)
Saved Process Window Contents (after - each pane saved individually)
