Peano 4
Loading...
Searching...
No Matches
peano4::parallel::Node Class Reference

Node is Peano's abstraction for the hybrid of MPI/threads. More...

#include <Node.h>

Collaboration diagram for peano4::parallel::Node:

Public Types

enum class  ExchangeMode { HorizontalData , VerticalData }
 
typedef std::pair< int, intPeriodicBoundaryStackIdentifier
 A periodic boundary stack is basically a stack (an integer), but I do augment it by a bitset which identifies which symmetry axis belong to this stack number.
 
typedef std::pair< int, MPI_Comm > GridDataExchangeMetaInformation
 

Public Member Functions

int getLocalTreeId (int treeId) const
 
int getGlobalTreeId (int treeId) const
 
virtual ~Node ()
 The standard destructor calls MPI_Finalize().
 
int getNumberOfRegisteredTrees () const
 
int reserveId (int rank, int forTreeId)
 This operation is not const as it does some internal bookkeeping.
 
int getRank (int treeId) const
 You hand in a tree number and the node tells you on which rank such a tree is hosted.
 
void deregisterId (int id)
 Only the SpacetreeSet should call this operation.
 
bool continueToRun ()
 You should call this operation only on the ranks >0 to find out whether you should do more iteration/sweeps.
 
void setNextProgramStep (int number)
 The user tells the set which program step to use, i.e.
 
int getCurrentProgramStep () const
 
void init ()
 
void shutdown ()
 The shutdown is invoked by peano4::shutdownSingletons()
 
GridDataExchangeMetaInformation getGridDataExchangeMetaInformation (int sendingTreeId, int receivingTreeId, ExchangeMode exchange) const
 I use two tags per spacetree per rank: one for boundary data (horizontal) and one for up-down and synchronous (forks) data exchanges (vertical and fork/join).
 

Static Public Member Functions

static void initMPIDatatypes ()
 I originally wanted to embed these guys into the singleton's constructor.
 
static void shutdownMPIDatatypes ()
 
static NodegetInstance ()
 This operation returns the singleton instance.
 
static bool isGlobalMaster (int treeId)
 Is this the global master?
 
static int getOutputStackNumberForHorizontalDataExchange (int id)
 Hand in a spacetree id and get back the number that we should use to send something to this tree.
 
static int getInputStackNumberForVerticalDataExchange (int id)
 Get the input stack where a tree writes all of its vertical data to/from when it exchanges information with id.
 
static int getOutputStackNumberForVerticalDataExchange (int id)
 
static int getInputStackNumberForHorizontalDataExchange (int id)
 Counterpart of getOutputStackNumberOfBoundaryExchange(int)
 
static std::set< PeriodicBoundaryStackIdentifiergetOutputStacksForPeriodicBoundaryExchange (const tarch::la::Vector< TwoPowerD, int > &flags)
 You hand in a the flags of a vertex and you get the boundary stack identifiers including their directions.
 
static std::string toString (const std::set< PeriodicBoundaryStackIdentifier > &data)
 
static int getOutputStackForPeriodicBoundaryExchange (int faceNumber)
 Identify output stack for periodic boundary data written by face.
 
static int getPeriodicBoundaryExchangeInputStackNumberForOutputStack (int outputStackNumber)
 Hand in an output stack number of a face and you get back the input stack number.
 
static PeriodicBoundaryStackIdentifier getPeriodicBoundaryExchangeInputStackNumberForOutputStack (PeriodicBoundaryStackIdentifier outputStackIdentifier)
 
static int mapPeriodicBoundaryExchangeOutputStackOntoInputStack (int outputStack)
 
static bool isHorizontalDataExchangeOutputStackNumber (int number)
 See getOutputStackNumberOfBoundaryExchange().
 
static bool isHorizontalDataExchangeInputStackNumber (int number)
 
static bool isVerticalDataExchangeOutputStackNumber (int number)
 
static bool isVerticalDataExchangeInputStackNumber (int number)
 
static bool isPeriodicBoundaryExchangeOutputStackNumber (int number)
 
static bool isStorageStackNumber (int number)
 There are communication stacks and storage stacks.
 
static int getTreeNumberTiedToExchangeStackNumber (int number)
 Gives you back the id of a communication partner, i.e.
 
static std::string toString (ExchangeMode mode)
 
static std::string getSemanticsForTag (int tag)
 I use this for debugging.
 

Static Public Attributes

static constexpr int Terminate = -2
 
static constexpr int MaxSpacetreesPerRank = 1024
 
static constexpr int UndefProgramStep = -1
 Value for _currentProgramStep.
 

Private Member Functions

 Node ()
 The standard constructor assigns the attributes default values and checks whether the program is compiled using the -DParallel option.
 
 Node (const Node &node)=delete
 The copy constructor is private.
 
int getId (int rank, int localTreeId) const
 Peano maps grid instance threads + mpi ranks onto global IDs through this routine.
 
void registerId (int id, int masterId)
 The operation is not thread-safe as we call it only internally, i.e.
 

Static Private Member Functions

static std::bitset< 2 *Dimensions > getPeriodicBoundaryNumber (const tarch::la::Vector< TwoPowerD, int > &flags)
 Analyse to which of the 2d faces it is adjacent.
 

Private Attributes

tarch::multicore::BooleanSemaphore _semaphore
 
MPI_Comm _dataExchangeCommunicators [MaxSpacetreesPerRank]
 
int _currentProgramStep
 
int _rankOrchestrationTag
 
std::map< int, TreeEntry_treeEntries
 

Static Private Attributes

static tarch::logging::Log _log
 Logging device.
 
static Node _singleton
 
static constexpr int StacksPerCommunicationPartner = 4
 We need one stack for outgoing data, one for incoming data, and we distinguish horizontal and vertical data flow.
 

Friends

class peano4::parallel::tests::NodeTest
 

Detailed Description

Node is Peano's abstraction for the hybrid of MPI/threads.

It represents one spacetree instance basically.

Author
Tobias Weinzierl

Definition at line 36 of file Node.h.

Member Typedef Documentation

◆ GridDataExchangeMetaInformation

Definition at line 382 of file Node.h.

◆ PeriodicBoundaryStackIdentifier

A periodic boundary stack is basically a stack (an integer), but I do augment it by a bitset which identifies which symmetry axis belong to this stack number.

So if we return (23,010), then data goes to stack 23 and it is a stack that does realise periodic boundary conditions along the y axis.

The second argument (as sketched above) is of type std::bitset<Dimensions>. Unfortunately, bitsets do not work properly within the pair type as they seem not to define the comparison operator properly. So I use an int here and wrap it into a bitset whenever necessary.

See also
toString()

Definition at line 216 of file Node.h.

Member Enumeration Documentation

◆ ExchangeMode

Enumerator
HorizontalData 
VerticalData 

Vertical data is data running from the master to the worker and the other way round.

Consequently, some tree management data are vertical data, too. See for example peano4::parallel::SpacetreeSet::addSpacetree().

Definition at line 370 of file Node.h.

Constructor & Destructor Documentation

◆ Node() [1/2]

peano4::parallel::Node::Node ( )
private

The standard constructor assigns the attributes default values and checks whether the program is compiled using the -DParallel option.

If this is not the case, a warning is logged.

Definition at line 61 of file Node.cpp.

◆ Node() [2/2]

peano4::parallel::Node::Node ( const Node & node)
privatedelete

The copy constructor is private.

◆ ~Node()

peano4::parallel::Node::~Node ( )
virtual

The standard destructor calls MPI_Finalize().

Definition at line 65 of file Node.cpp.

References assertionMsg, and tarch::mpi::Rank::getInstance().

Here is the call graph for this function:

Member Function Documentation

◆ continueToRun()

bool peano4::parallel::Node::continueToRun ( )

You should call this operation only on the ranks >0 to find out whether you should do more iteration/sweeps.

The spacetree set internally hijacks this operation also on rank 0 to trigger the send out of startup messages. Therefore, the operation is not const.

Definition at line 418 of file Node.cpp.

References tarch::mpi::Rank::getGlobalMasterRank(), tarch::mpi::Rank::getInstance(), tarch::mpi::Rank::getNumberOfRanks(), peano4::parallel::StartTraversalMessage::getStepIdentifier(), logDebug, logTraceIn, logTraceOutWith1Argument, peano4::parallel::StartTraversalMessage::receiveAndPollDanglingMessages(), peano4::parallel::StartTraversalMessage::sendAndPollDanglingMessages(), peano4::parallel::StartTraversalMessage::setStepIdentifier(), and peano4::parallel::StartTraversalMessage::toString().

Referenced by peano4::parallel::SpacetreeSet::traverse().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ deregisterId()

void peano4::parallel::Node::deregisterId ( int id)

Only the SpacetreeSet should call this operation.

See also
getId(int,int)

Definition at line 188 of file Node.cpp.

References assertion1, and logDebug.

Referenced by peano4::parallel::SpacetreeSet::cleanUpTrees().

Here is the caller graph for this function:

◆ getCurrentProgramStep()

int peano4::parallel::Node::getCurrentProgramStep ( ) const

Definition at line 447 of file Node.cpp.

Referenced by selectNextAlgorithmicStep(), and step().

Here is the caller graph for this function:

◆ getGlobalTreeId()

int peano4::parallel::Node::getGlobalTreeId ( int treeId) const

Definition at line 131 of file Node.cpp.

References tarch::mpi::Rank::getInstance(), tarch::mpi::Rank::getNumberOfRanks(), and tarch::mpi::Rank::getRank().

Here is the call graph for this function:

◆ getGridDataExchangeMetaInformation()

peano4::parallel::Node::GridDataExchangeMetaInformation peano4::parallel::Node::getGridDataExchangeMetaInformation ( int sendingTreeId,
int receivingTreeId,
ExchangeMode exchange ) const

I use two tags per spacetree per rank: one for boundary data (horizontal) and one for up-down and synchronous (forks) data exchanges (vertical and fork/join).

The nice thing about Peano's new data management is that it is stack-only. Furthermore, all stack data exchange is triggered via the spacetree set, i.e. the order of the stacks is known externally, too. So we effectively do not need that many tags even though we need different tags per tree pair.

Definition at line 476 of file Node.cpp.

References assertion, logTraceInWith3Arguments, logTraceOutWith1Argument, and toString().

Referenced by peano4::parallel::SpacetreeSet::exchangeAllHorizontalDataExchangeStacks(), peano4::parallel::SpacetreeSet::streamDataFromSplittingTreeToNewTree(), and peano4::parallel::tests::NodeTest::testTagCalculation().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ getId()

int peano4::parallel::Node::getId ( int rank,
int localTreeId ) const
private

Peano maps grid instance threads + mpi ranks onto global IDs through this routine.

To identify a global stack number, please use this function and add it to your local stack number.

Definition at line 113 of file Node.cpp.

References tarch::mpi::Rank::getInstance(), and tarch::mpi::Rank::getNumberOfRanks().

Here is the call graph for this function:

◆ getInputStackNumberForHorizontalDataExchange()

int peano4::parallel::Node::getInputStackNumberForHorizontalDataExchange ( int id)
static

◆ getInputStackNumberForVerticalDataExchange()

int peano4::parallel::Node::getInputStackNumberForVerticalDataExchange ( int id)
static

Get the input stack where a tree writes all of its vertical data to/from when it exchanges information with id.

Such vertical data is multiscale information or split/join data.

Definition at line 226 of file Node.cpp.

References peano4::grid::PeanoCurve::MaxNumberOfCoreStacksPerSpacetreeInstance.

Referenced by peano4::grid::Spacetree::receiveAndMergeGridVertexAtVerticalBoundary().

Here is the caller graph for this function:

◆ getInstance()

peano4::parallel::Node & peano4::parallel::Node::getInstance ( )
static

This operation returns the singleton instance.

Before using this instance, one has to call the init() operation on the instance returned.

Returns
The singleton instance

Definition at line 108 of file Node.cpp.

Referenced by peano4::parallel::SpacetreeSet::addSpacetree(), peano4::parallel::SpacetreeSet::answerQuestions(), peano4::parallel::SpacetreeSet::cleanUpTrees(), peano4::maps::HierarchicalStackMap< T >::clear(), peano4::maps::STDStackMap< T >::clear(), peano4::parallel::SpacetreeSet::deleteAllStacks(), peano4::maps::HierarchicalStackMap< T >::empty(), peano4::parallel::SpacetreeSet::exchangeAllHorizontalDataExchangeStacks(), peano4::parallel::SpacetreeSet::exchangeAllPeriodicBoundaryDataStacks(), peano4::maps::HierarchicalStackMap< T >::garbageCollection(), peano4::maps::STDStackMap< T >::garbageCollection(), peano4::parallel::SpacetreeSet::getAnswerTag(), peano4::maps::HierarchicalStackMap< T >::getForPop(), peano4::maps::HierarchicalStackMap< T >::getForPush(), peano4::maps::HierarchicalStackMap< T >::getKeys(), peano4::maps::HierarchicalStackMap< T >::holdsStack(), peano4::parallel::SpacetreeSet::init(), peano4::initSingletons(), main(), peano4::grid::Spacetree::receiveAndMergeGridVertexAtHorizontalBoundary(), peano4::grid::Spacetree::receiveAndMergeGridVertexAtVerticalBoundary(), peano4::grid::Spacetree::receiveAndMergeUserData(), runParallel(), runParallel(), selectNextAlgorithmicStep(), peano4::grid::Spacetree::sendGridVertex(), peano4::grid::Spacetree::sendUserData(), peano4::shutdownSingletons(), peano4::parallel::SpacetreeSet::split(), step(), peano4::parallel::SpacetreeSet::streamDataFromSplittingTreeToNewTree(), peano4::parallel::tests::NodeTest::testTagCalculation(), peano4::parallel::SpacetreeSet::traverse(), and peano4::grid::Spacetree::updateVertexAfterLoad().

Here is the caller graph for this function:

◆ getLocalTreeId()

◆ getNumberOfRegisteredTrees()

int peano4::parallel::Node::getNumberOfRegisteredTrees ( ) const
Returns
Number of registered trees on this rank

Definition at line 183 of file Node.cpp.

◆ getOutputStackForPeriodicBoundaryExchange()

int peano4::parallel::Node::getOutputStackForPeriodicBoundaryExchange ( int faceNumber)
static

Identify output stack for periodic boundary data written by face.

To make the mapping of face and vertex stacks consistent, we take the face number and determine what the adjacent, i.e. face-connected neighbour volume out of the \( 3^d \) adjacent, i.e. vertex-connected volumes is.

Take care that the result is not a mere linearisation of the index. You have to explicitly exclude the centre volume.

See also
getPeriodicBoundaryExchangeInputStackNumberForOutputStack() for a description how to compute an input stack index from the output index.
mapPeriodicBoundaryExchangeOutputStackOntoInputStack() for a mapping of periodic output stacks onto input stacks
Returns
The output stack for a periodic boundary data exchange for one face

Definition at line 250 of file Node.cpp.

References peano4::utils::dLinearised(), peano4::grid::PeanoCurve::MaxNumberOfCoreStacksPerSpacetreeInstance, peano4::grid::PeanoCurve::NumberOfPeriodicBoundaryConditionStacks, and ThreePowerD.

Referenced by peano4::grid::Spacetree::receiveAndMergeUserData(), and peano4::grid::Spacetree::sendUserData().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ getOutputStackNumberForHorizontalDataExchange()

int peano4::parallel::Node::getOutputStackNumberForHorizontalDataExchange ( int id)
static

Hand in a spacetree id and get back the number that we should use to send something to this tree.

Definition at line 211 of file Node.cpp.

References peano4::grid::PeanoCurve::MaxNumberOfCoreStacksPerSpacetreeInstance.

Referenced by peano4::parallel::SpacetreeSet::exchangeAllHorizontalDataExchangeStacks(), peano4::grid::Spacetree::sendGridVertex(), and peano4::grid::Spacetree::sendUserData().

Here is the caller graph for this function:

◆ getOutputStackNumberForVerticalDataExchange()

◆ getOutputStacksForPeriodicBoundaryExchange()

std::set< peano4::parallel::Node::PeriodicBoundaryStackIdentifier > peano4::parallel::Node::getOutputStacksForPeriodicBoundaryExchange ( const tarch::la::Vector< TwoPowerD, int > & flags)
static

You hand in a the flags of a vertex and you get the boundary stack identifiers including their directions.

Algorithm

We first analyse which faces with a periodic boundary flag are adjacent to the vertex. For this, we employ getPeriodicBoundaryNumber(). After that, we build the power set over the result: If a face is adjacent to the upper face and the left face, it also has to communicate along the diagonal, i.e. the logical combination of the two.

We now have to translate these powerset entries into stack indices. For this, we run over the \( 3^d \) neighbour cells or a cell, construct the index for this direction, and then look whether this index is contained within the power set.

Implementation

For the powerset, I'd like to use

 std::set< std::bitset<2*Dimensions> > powerSet;

but C++ defines no comparison operator on bitsets and therefore you can't use it in a set.

Definition at line 282 of file Node.cpp.

References assertion5, dfor3, enddforx, logTraceInWith2Arguments, logTraceOutWith2Arguments, peano4::grid::PeanoCurve::MaxNumberOfCoreStacksPerSpacetreeInstance, and peano4::grid::PeanoCurve::NumberOfPeriodicBoundaryConditionStacks.

Referenced by peano4::grid::Spacetree::receiveAndMergeGridVertexAtHorizontalBoundary(), peano4::grid::Spacetree::receiveAndMergeUserData(), peano4::grid::Spacetree::sendGridVertex(), peano4::grid::Spacetree::sendUserData(), and peano4::parallel::tests::NodeTest::testGetOutputStacksForPeriodicBoundaryExchange().

Here is the caller graph for this function:

◆ getPeriodicBoundaryExchangeInputStackNumberForOutputStack() [1/2]

int peano4::parallel::Node::getPeriodicBoundaryExchangeInputStackNumberForOutputStack ( int outputStackNumber)
static

Hand in an output stack number of a face and you get back the input stack number.

If you want to use it for merging data, you have to be careful: Think about the right face of your domain. It is assigned a BC output stack. This routine tells you onto which input stack this output stack is mapped. In return, if you read in the periodic BC data for the right face, you will merge a data with its own! You however have to merge it with the input BC stack from the left face.

This routine is not made to orchestrate the stack exchange! It is meant that you hand in number to which a vertex or face would write to. And then you get the number back which it should read from. This is not how the stacks are permuted after each step! If you want to achieve this, you have to call mapPeriodicBoundaryExchangeOutputStackOntoInputStack().

See also
peano4::parallel::SpacetreeSet::exchangeAllPeriodicBoundaryDataStacks() for a routine which uses this one in return.
peano4::grid::PeanoCurve:MaxNumberOfCoreStacksPerSpacetreeInstance for a discussion of the ordering of the stacks.

Definition at line 355 of file Node.cpp.

References assertion, logTraceInWith4Arguments, logTraceOutWith1Argument, peano4::grid::PeanoCurve::MaxNumberOfCoreStacksPerSpacetreeInstance, and peano4::grid::PeanoCurve::NumberOfPeriodicBoundaryConditionStacks.

Referenced by peano4::grid::Spacetree::receiveAndMergeGridVertexAtHorizontalBoundary(), peano4::grid::Spacetree::receiveAndMergeUserData(), and peano4::parallel::tests::NodeTest::testGetOutputStacksForPeriodicBoundaryExchange().

Here is the caller graph for this function:

◆ getPeriodicBoundaryExchangeInputStackNumberForOutputStack() [2/2]

◆ getPeriodicBoundaryNumber()

std::bitset< 2 *Dimensions > peano4::parallel::Node::getPeriodicBoundaryNumber ( const tarch::la::Vector< TwoPowerD, int > & flags)
staticprivate

Analyse to which of the 2d faces it is adjacent.

We enumerate again in the way we always enumerate, i.e. first the face with the x axis as normal, then the one with the y axis, ... And after these d faces, we enumerate all the opposite faces on the cube.

The routine is non-trivial: We store the \( 2^d \) adjacency flags per vertex, and now have to work backwards which periodic faces they do represent.

Definition at line 231 of file Node.cpp.

References dfor2, peano4::utils::dLinearised(), enddforx, k, and peano4::grid::Spacetree::RankOfPeriodicBoundaryCondition.

Referenced by peano4::parallel::tests::NodeTest::testGetPeriodicBoundaryNumber().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ getRank()

int peano4::parallel::Node::getRank ( int treeId) const

You hand in a tree number and the node tells you on which rank such a tree is hosted.

See also
getId(int,int)

Definition at line 119 of file Node.cpp.

References tarch::mpi::Rank::getInstance(), and tarch::mpi::Rank::getNumberOfRanks().

Referenced by peano4::parallel::SpacetreeSet::addSpacetree(), peano4::parallel::SpacetreeSet::cleanUpTrees(), peano4::parallel::SpacetreeSet::exchangeAllHorizontalDataExchangeStacks(), and peano4::parallel::SpacetreeSet::streamDataFromSplittingTreeToNewTree().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ getSemanticsForTag()

std::string peano4::parallel::Node::getSemanticsForTag ( int tag)
static

I use this for debugging.

When I have a tag, I can ask the node whether the node is aware what this tag actually means.

Definition at line 497 of file Node.cpp.

References assertion.

◆ getTreeNumberTiedToExchangeStackNumber()

int peano4::parallel::Node::getTreeNumberTiedToExchangeStackNumber ( int number)
static

Gives you back the id of a communication partner, i.e.

a spacetree number, which corresponds to stack number. This works for all kinds of stacks.

Definition at line 413 of file Node.cpp.

References peano4::grid::PeanoCurve::MaxNumberOfCoreStacksPerSpacetreeInstance.

Referenced by peano4::parallel::SpacetreeSet::exchangeAllHorizontalDataExchangeStacks().

Here is the caller graph for this function:

◆ init()

void peano4::parallel::Node::init ( )

Definition at line 77 of file Node.cpp.

References tarch::mpi::Rank::getInstance(), and tarch::mpi::Rank::reserveFreeTag().

Referenced by peano4::initSingletons().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ initMPIDatatypes()

void peano4::parallel::Node::initMPIDatatypes ( )
static

I originally wanted to embed these guys into the singleton's constructor.

However, the singleton might not yet be up when I run the (mpi) tests.

Definition at line 27 of file Node.cpp.

References peano4::datamanagement::CellMarker::initDatatype(), peano4::grid::AutomatonState::initDatatype(), peano4::grid::GridControlEvent::initDatatype(), peano4::grid::GridStatistics::initDatatype(), peano4::grid::GridVertex::initDatatype(), peano4::parallel::StartTraversalMessage::initDatatype(), peano4::parallel::TreeManagementMessage::initDatatype(), logTraceIn, and logTraceOut.

Referenced by peano4::initParallelEnvironment().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ isGlobalMaster()

bool peano4::parallel::Node::isGlobalMaster ( int treeId)
static

Is this the global master?

Definition at line 103 of file Node.cpp.

◆ isHorizontalDataExchangeInputStackNumber()

bool peano4::parallel::Node::isHorizontalDataExchangeInputStackNumber ( int number)
static

◆ isHorizontalDataExchangeOutputStackNumber()

bool peano4::parallel::Node::isHorizontalDataExchangeOutputStackNumber ( int number)
static

See getOutputStackNumberOfBoundaryExchange().

Definition at line 385 of file Node.cpp.

References peano4::grid::PeanoCurve::MaxNumberOfCoreStacksPerSpacetreeInstance.

Referenced by peano4::parallel::SpacetreeSet::exchangeAllHorizontalDataExchangeStacks().

Here is the caller graph for this function:

◆ isPeriodicBoundaryExchangeOutputStackNumber()

bool peano4::parallel::Node::isPeriodicBoundaryExchangeOutputStackNumber ( int number)
static

◆ isStorageStackNumber()

bool peano4::parallel::Node::isStorageStackNumber ( int number)
static

There are communication stacks and storage stacks.

This is a storage stack.

Definition at line 198 of file Node.cpp.

References peano4::grid::PeanoCurve::MaxNumberOfCoreStacksPerSpacetreeInstance.

Referenced by peano4::maps::HierarchicalStackMap< T >::garbageCollection(), and peano4::maps::STDStackMap< T >::garbageCollection().

Here is the caller graph for this function:

◆ isVerticalDataExchangeInputStackNumber()

bool peano4::parallel::Node::isVerticalDataExchangeInputStackNumber ( int number)
static

◆ isVerticalDataExchangeOutputStackNumber()

bool peano4::parallel::Node::isVerticalDataExchangeOutputStackNumber ( int number)
static

◆ mapPeriodicBoundaryExchangeOutputStackOntoInputStack()

◆ registerId()

void peano4::parallel::Node::registerId ( int id,
int masterId )
private

The operation is not thread-safe as we call it only internally, i.e.

you are not supposed to call this function directly.

See also
getId(int,int)

Definition at line 164 of file Node.cpp.

References assertion, assertion2, logTraceInWith2Arguments, logTraceOut, peano4::parallel::TreeEntry::setId(), and peano4::parallel::TreeEntry::setMaster().

Here is the call graph for this function:

◆ reserveId()

int peano4::parallel::Node::reserveId ( int rank,
int forTreeId )

This operation is not const as it does some internal bookkeeping.

It internally invokes registerId() on the result.

Parameters
forTreeIdIf we run a split, this is the rank that wants to split. In general, this always is the parent/master of the new rank. So if you move a rank, you have to ensure that the master is preserved.
Returns
-1 If there are no ids left anymore. In principle, we do not limit the number of threads at all. However, we have to work with a limited number of tags in MPI, so we put a cap on the number of trees per node.
See also
getId(int,int)

Definition at line 137 of file Node.cpp.

References tarch::mpi::Rank::getInstance(), logDebug, and logWarning.

Referenced by peano4::parallel::SpacetreeSet::answerQuestions(), and peano4::parallel::SpacetreeSet::split().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ setNextProgramStep()

void peano4::parallel::Node::setNextProgramStep ( int number)

The user tells the set which program step to use, i.e.

which adapter to use for the next grid run or which step to run next. Should only be called on global master.

Definition at line 441 of file Node.cpp.

References assertion1.

Referenced by runParallel(), runParallel(), and selectNextAlgorithmicStep().

Here is the caller graph for this function:

◆ shutdown()

void peano4::parallel::Node::shutdown ( )

The shutdown is invoked by peano4::shutdownSingletons()

If we run it on the global master, we set the program state to terminate and distribute this message. For this, we hijack the routine continueToRun(). This is really a hack. We in particular have to be careful and may not call the continue operation on the other ranks. They typically are stuck in a continue while loop in their main routine and just wait for the global master to send out this terminate command.

It is important that this is the very first call when you want your application to go down. Invoked on the global master, it triggers the shutdown on all the other ranks. You will get deadlocks if you try to free any common memory (RDMA) or free datatypes if you don't call the node's shutdown first.

Other ranks might struggle to receive the shutdown command as they might still ask for further subtrees, e.g., or issue load balancing. Therefore, I insert a Peano 4 barrier right after we've issued the shutdown. It makes the rank that goes down handle all incoming data until we can be sure that everybody is in shutdown mode.

Definition at line 452 of file Node.cpp.

References tarch::mpi::Rank::barrier(), tarch::mpi::Rank::getInstance(), tarch::services::ServiceRepository::getInstance(), logTraceIn, logTraceOut, tarch::services::ServiceRepository::receiveDanglingMessages(), and Terminate.

Referenced by peano4::shutdownSingletons().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ shutdownMPIDatatypes()

◆ toString() [1/2]

std::string peano4::parallel::Node::toString ( const std::set< PeriodicBoundaryStackIdentifier > & data)
static

Definition at line 262 of file Node.cpp.

Referenced by peano4::parallel::tests::NodeTest::testGetOutputStacksForPeriodicBoundaryExchange().

Here is the caller graph for this function:

◆ toString() [2/2]

std::string peano4::parallel::Node::toString ( ExchangeMode mode)
static

Definition at line 92 of file Node.cpp.

Friends And Related Symbol Documentation

◆ peano4::parallel::tests::NodeTest

friend class peano4::parallel::tests::NodeTest
friend

Definition at line 46 of file Node.h.

Field Documentation

◆ _currentProgramStep

int peano4::parallel::Node::_currentProgramStep
private

Definition at line 69 of file Node.h.

◆ _dataExchangeCommunicators

MPI_Comm peano4::parallel::Node::_dataExchangeCommunicators[MaxSpacetreesPerRank]
private

Definition at line 67 of file Node.h.

◆ _log

tarch::logging::Log peano4::parallel::Node::_log
staticprivate

Logging device.

Definition at line 51 of file Node.h.

◆ _rankOrchestrationTag

int peano4::parallel::Node::_rankOrchestrationTag
private

Definition at line 71 of file Node.h.

◆ _semaphore

tarch::multicore::BooleanSemaphore peano4::parallel::Node::_semaphore
private

Definition at line 55 of file Node.h.

◆ _singleton

peano4::parallel::Node peano4::parallel::Node::_singleton
staticprivate

Definition at line 53 of file Node.h.

◆ _treeEntries

std::map< int, TreeEntry > peano4::parallel::Node::_treeEntries
private

Definition at line 73 of file Node.h.

◆ MaxSpacetreesPerRank

constexpr int peano4::parallel::Node::MaxSpacetreesPerRank = 1024
staticconstexpr

◆ StacksPerCommunicationPartner

constexpr int peano4::parallel::Node::StacksPerCommunicationPartner = 4
staticconstexprprivate

We need one stack for outgoing data, one for incoming data, and we distinguish horizontal and vertical data flow.

Definition at line 62 of file Node.h.

Referenced by peano4::parallel::tests::NodeTest::testTagCalculation().

◆ Terminate

constexpr int peano4::parallel::Node::Terminate = -2
staticconstexpr

Definition at line 38 of file Node.h.

Referenced by shutdown().

◆ UndefProgramStep

constexpr int peano4::parallel::Node::UndefProgramStep = -1
staticconstexpr

Value for _currentProgramStep.

Definition at line 44 of file Node.h.


The documentation for this class was generated from the following files: