PSLDU(3S)PSLDU(3S)NAME
PSLDU_Destroy, PSLDU_ExtractPerm, PSLDU_Factor, PSLDU_FactorOOC,
PSLDU_OOCLimit, PSLDU_OOCPath, PSLDU_Ordering, PSLDU_Preprocess,
PSLDU_PreprocessZ, PSLDU_Solve, PSLDU_SolveM, PSLDU_Storage - Parallel
sparse unsymmetric linear system solver
SYNOPSIS
Fortran synopsis:
SUBROUTINE PSLDU_DESTROY (token)
INTEGER token
SUBROUTINE PSLDU_EXTRACTPERM (token, perm)
INTEGER token, perm(*)
SUBROUTINE PSLDU_FACTOR (token, n, pointers, indices, values)
INTEGER token, n, pointers(*), indices(*)
DOUBLE PRECISION values(*)
SUBROUTINE PSLDU_FACTOROOC (token, n, pointers, indices, values)
INTEGER token, n, pointers(*), indices(*)
DOUBLE PRECISION values(*)
SUBROUTINE PSLDU_OOCLIMIT (token, ooclimit)
INTEGER token
DOUBLE PRECISION ooclimit
SUBROUTINE PSLDU_OOCPATH (token, oocpath)
INTEGER token
CHARACTER oocpath(*)
SUBROUTINE PSLDU_ORDERING (token, method)
INTEGER token, method
SUBROUTINE PSLDU_PREPROCESS (token, n, pointers, indices, non_zeros,
ops)
INTEGER token, n, pointers(*), indices(*)
INTEGER*8 non_zeros
DOUBLE PRECISION ops
SUBROUTINE PSLDU_PREPROCESSZ (token, n, pointers, indices, mask,
non_zeros, ops)
INTEGER token, n, pointers(*), indices(*), mask(*)
INTEGER*8 non_zeros
DOUBLE PRECISION ops
SUBROUTINE PSLDU_SOLVE (token, x, b)
INTEGER token
DOUBLE PRECISION x(*), b(*)
Page 1
PSLDU(3S)PSLDU(3S)
SUBROUTINE PSLDU_SOLVEM (token, X, B, nrhs)
INTEGER token, nrhs
DOUBLE PRECISION X(*), B(*)
DOUBLE PRECISION FUNCTION PSLDU_STORAGE(token)
INTEGER token
C/C++ synopsis:
#include <scsl_sparse.h>
void PSLDU_Destroy (int token );
void PSLDU_ExtractPerm (int token, int perm[] );
void PSLDU_Factor (int token, int n, int pointers[], int indices[],
double values[] );
void PSLDU_FactorOOC (int token, int n, int pointers[], int
indices[], double values[] );
void PSLDU_OOCLimit (int token, double ooclimit );
void PSLDU_OOCPath (int token, char oocpath[] );
void PSLDU_Ordering (int token, int method );
void PSLDU_Preprocess (int token, int n, int pointers[], int
indices[], long long *non_zeros, double *ops );
void PSLDU_PreprocessZ (int token, int n, int pointers[], int
indices[], int mask[], long long *non_zeros, double *ops );
void PSLDU_Solve (int token, double x[], double b[] );
void PSLDU_SolveM (int token, double X[], double B[], int nrhs);
double PSLDU_Storage (int token);
IMPLEMENTATION
These routines are part of the SCSL Scientific Library and can be loaded
using either the -lscs or the -lscs_mp option. The -lscs_mp option
directs the linker to use the multi-processor version of the library.
When linking to SCSL with -lscs or -lscs_mp, the default integer size is
4 bytes (32 bits). Another version of SCSL is available in which integers
are 8 bytes (64 bits). This version allows the user access to larger
memory sizes and helps when porting legacy Cray codes. It can be loaded
by using the -lscs_i8 option or the -lscs_i8_mp option. A program may
use only one of the two versions; 4-byte integer and 8-byte integer
library calls cannot be mixed.
Page 2
PSLDU(3S)PSLDU(3S)
The C and C++ prototypes shown above are appropriate for the 4-byte
integer version of SCSL. When using the 8-byte integer version, the
variables of type int become long long and the <scsl_sparse_i8.h> header
file should be included.
DESCRIPTION
NOTE: these interfaces are obsolete and will no longer be supported in
future versions of SCSL. Please use the routines described in the DPSLDU
man page instead. For complex data types, see the ZPSLDU man page.
PSLDU solves sparse unsymmetric linear systems of the form Ax = b where A
is an n-by-n input matrix having symmetric non-zero pattern but
unsymmetric non-zero values, b is an input vector of length n, and x is
an unknown vector of length n.
PSLDU uses a direct method. A is factored into the following form:
A = L D U
where L is a lower triangular matrix with unit diagonal, D is a diagonal
matrix, and U is an upper triangular matrix with unit diagonal.
Note that NO PIVOTING FOR STABILITY is performed during factorization.
The PSLDU library contains five main routines.
* PSLDU_Ordering() allows the user to select one of five possible
reordering methods to be used in the matrix preprocessing phase.
* PSLDU_Preprocess() performs preprocessing operations on the structure
of A (heuristic reordering to reduce fill in L and U, symbolic
factorization, etc.).
* PSLDU_Factor() factors the matrix A into L and U, using the
previously computed preprocessing data.
* PSLDU_Solve() solves for a vector x, given an input vector b.
* PSLDU_Destroy() frees all storage associated with the matrix A
(including L, D, U, and various data structures computed during
preprocessing).
The user can call PSLDU_Factor() several times after a single call to
PSLDU_Preprocess() to factor multiple matrices with identical non-zero
structures but different values. Similarly, the user can call
PSLDU_Solve() several times after a single call to PSLDU_Factor() to
solve for multiple right-hand-sides. Also, the user can call
PSLDU_SolveM() to solve for multiple right-hand-sides all stored in a
single array.
Page 3
PSLDU(3S)PSLDU(3S)
Sparse Matrix Format
Sparse matrix A must be input to PSLDU in Harwell-Boeing format (also
known as Compressed Column Storage format).
The matrix is held in three arrays: pointers, indices, and values. The
indices array contains the row indices of the non-zeros in A. The values
array holds the corresponding non-zero values. The pointers array
contains the index in indices for the first non-zero in each column of A.
Thus, the row indices for the non-zeros in column i can be found in
locations indices[pointers[i]] through indices[pointers[i+1]-1]. The
corresponding values can be found in location values[pointers[i]] through
values[pointers[i+1]-1].
PSLDU imposes one constraint on the representation of the A matrix. The
non-zeros within each column must appear in order of increasing row
number.
In the following example, the unsymmetric matrix
1.0 0.0 5.0 0.0
0.0 3.0 0.0 8.0
2.0 0.0 7.0 0.0
0.0 4.0 0.0 9.0
would be represented in FORTRAN as follows:
INTEGER pointers(5), indices(8), i
DOUBLE PRECISION values(8)
DATA (pointers(i), i = 1, 5) / 1, 3, 5, 7, 9 /
DATA (indices(i), i = 1, 8) / 1, 3, 2, 4, 1, 3, 2, 4 /
DATA (values(i), i = 1, 8) / 1.0, 2.0, 3.0, 4.0, 5.0,
& 7.0, 8.0, 9.0 /
Zero-based indexing is used in C, so the pointers, indices, and values
arrays would contain the following:
int pointers[] = {0, 2, 4, 6, 8};
int indices[] = {0, 2, 1, 3, 0, 2, 1, 3};
double values[] = {1.0, 2.0, 3.0, 4.0, 5.0, 7.0, 8.0, 9.0};
Ordering Methods
The PSLDU_Ordering(token, method) routine allows the user to change the
ordering method used to pre-order the matrix before factorization. This
routine must be called before calling PSLDU_Preprocess. Five options are
currently available for the method parameter:
* Method 0 performs no pre-ordering
Page 4
PSLDU(3S)PSLDU(3S)
* Method 1 performs Approximate Minimum Fill ordering
* Method 2 performs a single nested dissection ordering (default).
This method is often called "Extreme matrix ordering".
* Method 3 performs multiple nested dissection orderings (in parallel)
* Method 4 performs multiple nested dissection (the same as in Method
3), but it uses a feedback file to "learn" from the previous solves
of the same matrix structure and it performs more orderings. The
multiple nested dissection technique of Methods 3 and 4 is also
referred to as "Extreme2 matrix ordering".
Method 2 is significantly more expensive than Method 1, but it usually
produces significantly better orderings. Method 3 is especially
effective on multi-processor systems. It computes OMP_NUM_THREADS (where
OMP_NUM_THREADS is an environment variable indicating the number of
processors to be used for parallel computation) matrix orderings using
different starting points for the algorithm and uses the ordering that
will lead to the fewest floating-point operations to factorize the
matrix.
Method 4 is useful only when the same non-zero structure is used for
multiple solves. Method 4 keeps a record in a "feedback" file of a
signature for non-zero structures for a maximum of 200 matrices and of
the starting point that was saved from a previous solve for that
structure. In the next Method 4 ordering for that non-zero structure,
that best starting point and 2 * OMP_NUM_THREADS - 1 new ones generate
orderings. The best ordering is used. In this way, the quality of
orderings stay the same or improve over time.
Methods 3 and 4 typically take more time for the matrix preprocessing
than the default. However, on large systems or on repeated
factorizations, significant overall speedups (1.1X to 2X) can be obtained
compared to Method 2.
Extracting the permutation vector
Unless ordering Method 0 is used, PSLDU applies a symmetric permutation
to matrix A before the factorization step; the resulting permuted matrix
generally has significantly less fill-in than the original matrix. The
user can obtain the permutation matrix associated with a given token by
calling PSLDU_ExtractPerm(token, perm). The permutation is returned as an
integer array of length n, with 1 <= perm(i) <= n (0 <= perm[i] < n for C
code).
A value of k for perm(i) implies that node k in the original ordering is
node i in the new ordering.
Matrices with zeros on the diagonal
As noted above, no pivoting is done for stability during factorization;
when zero or near-zero pivots are encountered, PSLDU usually fails. In
these cases, it may be possible to use PSLDU_PreprocessZ() to obtain a
Page 5
PSLDU(3S)PSLDU(3S)
slightly different, but stable, ordering. The user provides an
additional integer array, mask, as an argument to PSLDU_PreprocessZ().
If mask(i)=0, then PSLDU will attempt to maximize the diagonal element
|Aii|.
Memory usage
The returned value of PSLDU_Storage() is an estimate of the amount of
storage required (in millions of bytes) by the solver's data structures
for a given matrix system.
Out-of-core Factorization
The storage associated with the factor can be managed in two ways. The
PSLDU_Factor() routine allocates memory for the factor and manages it
internally, releasing it only when PSLDU_Destroy() is called. The
alternative is to do out-of-core factorization by calling
PSLDU_FactorOOC(). This routine uses a small amount of in-core memory,
placing the remainder of the factor matrix on disk as it is computed.
The user can call PSLDU_OOCPath() to indicate the directory in which the
factor file should be written, and PSLDU_OOCLimit() to indicate how much
memory to use to hold portions of the factor matrix in-core. More in-
core memory generally leads to less disk I/O and higher performance
during the factorization. The only required change is to move from in-
core factorization to out-of-core factorization is the change from
PSLDU_Factor() to PSLDU_FactorOOC(). The other routines (PSLDU_Solve(),
PSLDU_Destroy(), etc.) handle out-of-core factors transparently. Note
that PSLDU_FactorOOC and subsequent calls to PSLDU_Solve are not
parallelized.
Multiple Right-Hand-Sides
PSLDU can solve for large numbers of right-hand-sides with one call to
PSLDU_SolveM(). It solves these right hand sides in parallel, with each
processor solving up to four at a time.
Arguments
These routines have the following arguments:
token (input) PSLDU can handle multiple matrices simultaneously. The
token distinguishes between active matrices. The token passed
to PSLDU_Factor() must match the token used in some previous
call to PSLDU_Preprocess(). Similarly, the token passed to
PSLDU_Solve() must match the token used in some previous call
to PSLDU_Factor(). 0 <= token <= 19.
method (input) An integer specifying the ordering method used during
preprocessing. 0 <= method <= 4.
n (input) The number of rows and columns in the matrix A. n >=
0.
pointers, indices, values
(input) The pointers and indices arrays store the non-zero
structure of sparse input matrix A in Harwell-Boeing or
Page 6
PSLDU(3S)PSLDU(3S)
Compressed Sparse Column (CSC) format.
The pointers array stores n+1 integers, where pointers[i] gives
the index in indices of the first non-zero in column i of A.
The indices array stores the row indices of the non-zeros in A.
The values array stores the non-zero values in the matrix A.
non_zeros (output) The number of non-zero values in L and U.
ops (output) The number of floating-point operations required to
factor A.
mask (input) An integer array of length n used in
PSLDU_PreprocessZ(). If mask(i)=0, then node i of matrix A is
ordered after all of its neighbors in an attempt to avoid a
zero pivot.
b (input) The right-hand-side vector in a PSLDU_Solve call.
x (output) The solution vector in a PSLDU_Solve call.
nrhs (input) The number of right-hand side vectors present in a
PSLDU_SolveM() call.
B (input) The right-hand-side matrix in a PSLDU_SolveM() call.
Must be stored in column-major order, and each of the nrhs
columns must have length n.
X (output) The solution matrix in a PSLDU_SolveM() call. Must be
stored in column-major order, and each of the nrhs columns must
have length n.
oocpath (input) A character array/string with a path to the directory
where the temporary out-of-core factor files should be stored.
If this path is on a striped (or raid-0) file system, the
performance of the out-of-core solves can be considerably
improved. The default path is /usr/tmp.
ooclimit (input) A double precision number indicating the number of
Mbytes of random access memory that should be used for factor
storage during a call to PSLDU_FactorOOC. Note that there are
many other arrays used besides those directly used to store the
factorization, so total RAM usage by the solve will exceed this
number. The default is 64 MB.
perm (output) An integer array of length n containing the
permutation used to reorder matrix A.
ENVIRONMENT VARIABLES
Two environment variables can affect the operation of ordering methods 3
and 4. SPARSE_NUM_ORDERS can be used to change the number of orderings
performed from the default of OMP_NUM_THREADS for Method 3 and
Page 7
PSLDU(3S)PSLDU(3S)
(2*OMP_NUM_THREADS) for Method 4. SPARSE_FEEDBACK_FILE can be set to the
path and file name where the feedback information will be kept;
otherwise, the default feedback file is $HOME/.sparseFeedback. This file
will be less than 5K bytes.
The environment variable OMP_NUM_THREADS determines the number of
processors that are used for the numerical factorization. The out-of-core
solve is limited to one processor. Setting the environment variable
PSLDU_VERBOSE causes PSLDU to output information about the factorization.
NOTES
These routines are optimized and parallelized for the SGI R8000 and
R1x000 platforms.
SEE ALSOINTRO_SCSL(3S), INTRO_SOLVERS(3S), DPSLDLT(3S), ZPSLDLT(3S), DPSLDU(3S),
ZPSLDU(3S)
Page 8
