org.javaseis.parallel
Class DistributedArray

java.lang.Object
  org.javaseis.array.MultiArray
      org.javaseis.parallel.DistributedArray

All Implemented Interfaces:

IMultiArray

public class DistributedArray
extends MultiArray

This class provides support for "distributed parallel arrays", a multidimensional array that has one or more dimensions spread across mulitple tasks in a parallel parallel context. The design follows the concepts from High Performance Fortran as developed by Ken Kennedy and others at Rice University in the 1990s.

The most usable and performant layout is the case where the last dimension of a multidimensional Fortran array is spread across processors. In the 2D case, "strips" of the 2D array are "owned" by each task in a parallel context. In the 3D case, "slabs" of data are owned by each task.

Decompositions are defined by the Decomposition class, which provides methods for obtaining appropriate index ranges and padded axis lengths.

When a Distributed MultiArray is allocated, distributed lengths are padded out to a factor of the number of nodes to accomodate the requested decomposition. Programs that use this class will need to account for this in the same manner as using a Fourier Transform package, for example.

Author:

Chuck Mosher for JavaSeis.org

Field Summary

(package private) int	_nParDims
(package private) IParallelContext	_parallelContext
(package private) Decomposition	_parDecomp
(package private) int	_pdim

Fields inherited from class org.javaseis.array.MultiArray

_arrayLength, _bufLength, _classType, _elementCount, _frameLength, _hypercubeLength, _lengths, _maxIndex, _ndim, _offset, _offsetMultiplier, _traceLength, _volumeLength, bytes, doubles, floats, ints, longs, objects, shorts

Constructor Summary

DistributedArray()
Constuct an empty DistributedArray

DistributedArray(int ndim, java.lang.Class classType, int[] lengths)

DistributedArray(IParallelContext pc, java.lang.Class classType, int ndim, int elementCount, int[] lengths, int decompType)
Construct a parallel distributed array with the last dimension distributed

DistributedArray(IParallelContext pc, java.lang.Class classType, int ndim, int elementCount, int[] lengths, int[] decompTypes)
Construct a parallel distributed array padded for potential transposes

Method Summary

static MultiArray	factory(int ndim, java.lang.Class classType, int elementCount, int[] lengths)
static MultiArray	float2D(int n1, int n2)
static MultiArray	float3D(int n1, int n2, int n3)
int[]	getActiveSection()
int	getGlobalLength(int index)
int	getLength(int index) Return the length of a particular dimension
int	getLocalLength(int index)
int[]	getLocalShape()
IParallelContext	getParallelContext()
int[]	getShape() Return the length of each dimension in this MultiArray
int	index(int[] position) Return the index in the underlying storage of the data at a particular position in the MultiArray
int	index(int[] position, int element) Return the index in the underlying storage of an element at a particular position in the MultiArray.
int	indexFortran(int[] position) Return the index in the underlying storage of the data at a particular position in the MultiArray.
int	indexFortran(int[] position, int element) Return the index in the underlying storage of an element at a particular position in the MultiArray.
int	localToGlobal(int dimension, int index)
static void	main(java.lang.String[] args)
void	printContents()
int[]	range(int[] position, int dimension) Return a zero-based index triplet defining start,end,increment that will traverse a specified dimension for this MultiArray.
int[]	range(int[] position, int dimension, int start, int end, int increment) Return a zero-based index triplet defining start,end,increment that can be used to access the elements of the underlying 1D array storage for this MultiArray.
int[]	range(int[] position, int dimension, int start, int end, int increment, int element) Return a zero-based index triplet defining start,end,increment that can be used to access individual elements of the underlying 1D array storage for this MultiArray.
void	setActiveSection(int[] lengths)
void	setShape(int[] newLengths) Set the shape of this MultiArray, in Fortran style with the "fast" dimension first
void	tran132() 3D distributed transpose Uses same approach as 2D distributed transpose, but moves vectors rather than individual elements.
<T> void	tran132(T[] buf) Generic method for object array transpose
void	tran21() Transpose a 2D distributed array, where the 2nd dimension is spread across processors.
<T> void	tran21(T[] buf) Generic method for object array transpose
void	tran213() Transpose the first two dimensions of a 3D DistributedArray If the number of elements per sample is not 1, the transpose is "promoted" to tran1324 using the element count as the first dimension.
void	transpose(TransposeType type) Apply the requested transpose to this MultiArray

Methods inherited from class org.javaseis.array.MultiArray

allocate, bytes, clone, deallocate, doubles, factory, floats, getDimensions, getElementCount, getFrame, getFrame, getFrame, getFrame, getFrame, getFrame, getFrame, getOffset, getTrace, getTrace, getTrace, getTrace, getTrace, getTrace, getTrace, ints, longs, objects, putFrame, putFrame, putFrame, putFrame, putFrame, putFrame, putFrame, putTrace, putTrace, putTrace, putTrace, putTrace, putTrace, putTrace, rangeFortran, rangeFortran, rangeFortran, setArray, setArray, setArray, setArray, setArray, setArray, setArray, setClassType, setDimensions, setElementCount, setOffset, shorts, tran1243

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

_nParDims

int _nParDims

_parallelContext

IParallelContext _parallelContext

_parDecomp

Decomposition _parDecomp

_pdim

int _pdim

Constructor Detail

DistributedArray

public DistributedArray()

Constuct an empty DistributedArray

DistributedArray

public DistributedArray(int ndim,
                        java.lang.Class classType,
                        int[] lengths)

DistributedArray

public DistributedArray(IParallelContext pc,
                        java.lang.Class classType,
                        int ndim,
                        int elementCount,
                        int[] lengths,
                        int decompType)

Construct a parallel distributed array with the last dimension distributed

Parameters:

pc - parallel context for the distributed array

classType - class type for elements, i.e. float.class, MyObject.class, etc.

ndim - number of dimensions of the parallel array

lengths - length of each dimension of the array

decompType - decomposition type, Decomposition.BLOCK or Decomposition.CIRCULAR

DistributedArray

public DistributedArray(IParallelContext pc,
                        java.lang.Class classType,
                        int ndim,
                        int elementCount,
                        int[] lengths,
                        int[] decompTypes)

Construct a parallel distributed array padded for potential transposes

Parameters:

pc - parallel context for the distributed array

classType - class type for elements, i.e. float.class, MyObject.class, etc.

ndim - number of dimensions of the parallel array

elementCount - number of elements per sample, i.e. 2 for complex numbers

lengths - input length of each dimension of the array

decompTypes - array of decomposition types for each axis, NONE, BLOCK, or CIRCULAR

Method Detail

factory

public static MultiArray factory(int ndim,
                                 java.lang.Class classType,
                                 int elementCount,
                                 int[] lengths)

float2D

public static MultiArray float2D(int n1,
                                 int n2)

float3D

public static MultiArray float3D(int n1,
                                 int n2,
                                 int n3)

getActiveSection

public int[] getActiveSection()

getGlobalLength

public int getGlobalLength(int index)

getLength

public int getLength(int index)

Description copied from interface: IMultiArray

Return the length of a particular dimension

Specified by:

getLength in interface IMultiArray

Overrides:

getLength in class MultiArray

Parameters:

index - the index of the dimension whose length will be returned

Returns:

length of the dimension

getLocalLength

public int getLocalLength(int index)

getLocalShape

public int[] getLocalShape()

getParallelContext

public IParallelContext getParallelContext()

getShape

public int[] getShape()

Description copied from interface: IMultiArray

Return the length of each dimension in this MultiArray

Specified by:

getShape in interface IMultiArray

Overrides:

getShape in class MultiArray

Returns:

length of each dimension

index

public int index(int[] position)

Description copied from interface: IMultiArray

Return the index in the underlying storage of the data at a particular position in the MultiArray

Specified by:

index in interface IMultiArray

Overrides:

index in class MultiArray

Parameters:

position - position of the sample to be accessed using zero-based indices

Returns:

zero-based index in the underlying storage of this sample

index

public int index(int[] position,
                 int element)

Description copied from interface: IMultiArray

Return the index in the underlying storage of an element at a particular position in the MultiArray. This method is used for Arrays where the SeisDataType defines multiple "elements" for each "sample" (i.e. complex or multicomponent data)

Specified by:

index in interface IMultiArray

Overrides:

index in class MultiArray

Parameters:

position - position of the sample to be accessed using zero-based indices

element - index of the "sample" "element" that will be accessed

Returns:

zero-based index in the underlying storage of the element

indexFortran

public int indexFortran(int[] position)

Description copied from interface: IMultiArray

Return the index in the underlying storage of the data at a particular position in the MultiArray. This method uses Fortran style 1 to N based indexing.

Specified by:

indexFortran in interface IMultiArray

Overrides:

indexFortran in class MultiArray

Parameters:

position - position of the sample to be accessed using Fortran indices

Returns:

Fortran index in the underlying storage of this sample

indexFortran

public int indexFortran(int[] position,
                        int element)

Description copied from interface: IMultiArray

Return the index in the underlying storage of an element at a particular position in the MultiArray. This method uses Fortran style 1 to N based indexing.

Specified by:

indexFortran in interface IMultiArray

Overrides:

indexFortran in class MultiArray

Parameters:

position - position of the sample to be accessed using Fortran indices

element - index of the "sample" "element" that will be accessed

Returns:

Fortran index in the underlying storage of the element

localToGlobal

public int localToGlobal(int dimension,
                         int index)

main

public static void main(java.lang.String[] args)

Parameters:

args -

Throws:

MPIException

printContents

public void printContents()

range

public int[] range(int[] position,
                   int dimension)

Description copied from interface: IMultiArray

Return a zero-based index triplet defining start,end,increment that will traverse a specified dimension for this MultiArray. If the SeisDataType defines "samples" that are arrays (i.e. complex or multicomponent samples) the range points to the first element of each "sample". The dimension argument defines which dimension will be traversed. The corresponding element of the "position" argument is ignored (in favor of the "start" value).

Specified by:

range in interface IMultiArray

Overrides:

range in class MultiArray

Parameters:

position - starting position in Fortran style indexing

dimension - the dimension that will be traversed

Returns:

zero based index triplet for the requested dimesion

range

public int[] range(int[] position,
                   int dimension,
                   int start,
                   int end,
                   int increment)

Description copied from interface: IMultiArray

Return a zero-based index triplet defining start,end,increment that can be used to access the elements of the underlying 1D array storage for this MultiArray. If the SeisDataType defines "samples" that are arrays (i.e. complex or multicomponent samples) the range points to the first element of each "sample". The dimension argument defines which Fortran dimension will be traversed. The corresponding element of the "position" argument is ignored (in favor of the "start" value).

Specified by:

range in interface IMultiArray

Overrides:

range in class MultiArray

Parameters:

position - starting position in Fortran style indexing

dimension - the dimension that will be traversed

start - start index along the requested dimension

end - ending index

increment - index increment

Returns:

offset triplet for the requested range

range

public int[] range(int[] position,
                   int dimension,
                   int start,
                   int end,
                   int increment,
                   int element)

Description copied from interface: IMultiArray

Return a zero-based index triplet defining start,end,increment that can be used to access individual elements of the underlying 1D array storage for this MultiArray. This method is used to access the individual "elements" of "samples" that contain multiple values (i.e. complex or multicomponent).

Specified by:

range in interface IMultiArray

Overrides:

range in class MultiArray

Parameters:

position - starting position in Fortran style indexing

dimension - the dimension that will be traversed

start - start index along the requested dimension

end - ending index

increment - index increment

element - the index of the element that will be accessed for the range selection

Returns:

offset triplet for the requested range

setActiveSection

public void setActiveSection(int[] lengths)

setShape

public void setShape(int[] newLengths)

Description copied from interface: IMultiArray

Set the shape of this MultiArray, in Fortran style with the "fast" dimension first

Specified by:

setShape in interface IMultiArray

Overrides:

setShape in class MultiArray

Parameters:

newLengths - length of each dimension

tran132

public void tran132()

3D distributed transpose Uses same approach as 2D distributed transpose, but moves vectors rather than individual elements. To visualize this transpose, apply the same idea of splitting axes into strips and tiles:

 Start with a 3D array with the third dimension spread across processors:
        n1 . n2 . n3/p | p
         x    y     z1  z2
 Divide the second and third (y and z respectively) dimensions by
 the number of processors (p) to divide into tiles:
        n1 . n2/p . p . n3/p | p
         x     y1  y2     z1  z2                                             
 Use local transpose to exchange y2 and z1, leaving z2 unchanged                    
        n1 . n2/p . n3/p . p | p
         x     y1     z1  y2  z2
 Transpose y2 and z2 across processors
        n1 . n2/p . n3/p . p | p
         x     y1     z1  z2  y2
 Collapse z1 and z2 back to z                                                                         
        n1 . n2/p . n3 | p
         x     y1    z  y2                                                                               
 Locally transpose y1 and z
        n1 . n3 . n2/p | p
         x    z     y1  y2
 Completed transpose has y spread across processors
 

Overrides:

tran132 in class MultiArray

tran132

public <T> void tran132(T[] buf)

Generic method for object array transpose

Type Parameters:

T -

Parameters:

buf - object array matching MultiArray type with min length n1/p * n2/p

tran21

public void tran21()

Transpose a 2D distributed array, where the 2nd dimension is spread across processors. In following comments, 'p' is the number of processors in the parallel context.

  We have 3 transpose operations available:
   tran21 - transpose local (n1,n2) to (n2,n1)
   tran132 - transpose local (n1,n2,n3) to (n1,n3,n2)
   ttran - transpose last two dimensions of distributed array
            (n1,p,p), where p is the number of processors.

  Treat 2D array as a 3D array distributed over the last dimension.
  This can be thought of as 'strips' on each processor.
         n1 . n2  ->  n1 . n2/p | p
          x    y       x     y1  y2
  Use local transpose to exchange x and y1:
         n1 . n2/p | p  ->  n2/p . n1 | p
          x     y1  y2        y1    x  y2
  Now expand the x direction, which can be thought of as
  creating a set of 'tiles' on each processor. Use 'ttran' to
  transpose tiles across processors:
         n2/p . n1/p . p | p  ->  n2/p . n1/p . p | p
           y1     x1  x2  y2        y1     x1  y2  x2
  Then use tran132 to collect y direction:
         n2/p . n1/p . p | p  ->  n2/p . p . n1/p | p
           y1     x1  y2  x2        y1  y2     x1  x2
  Collapse y to yield transposed array:
         n2 . n1/p | p
          y     x1  x2

Overrides:

tran21 in class MultiArray

tran21

public <T> void tran21(T[] buf)

Generic method for object array transpose

Type Parameters:

T -

Parameters:

buf - object array matching MultiArray type with min length n1/p * n2/p

tran213

public void tran213()

Transpose the first two dimensions of a 3D DistributedArray If the number of elements per sample is not 1, the transpose is "promoted" to tran1324 using the element count as the first dimension.

transpose

public void transpose(TransposeType type)

Apply the requested transpose to this MultiArray

Specified by:

transpose in interface IMultiArray

Overrides:

transpose in class MultiArray

Parameters:

type - the type of transpose to apply