 |
Peano
|
|
Peano
|
Node is Peano's abstraction for the hybrid of MPI/threads. More...
#include <Node.h>
Public Types | |
| enum class | ExchangeMode { HorizontalData , VerticalData } |
| typedef std::pair< int, int > | PeriodicBoundaryStackIdentifier |
| A periodic boundary stack is basically a stack (an integer), but I do augment it by a bitset which identifies which symmetry axis belong to this stack number. | |
| typedef std::pair< int, MPI_Comm > | GridDataExchangeMetaInformation |
Public Member Functions | |
| int | getLocalTreeId (int treeId) const |
| int | getGlobalTreeId (int treeId) const |
| virtual | ~Node () |
| The standard destructor calls MPI_Finalize(). | |
| int | getNumberOfRegisteredTrees () const |
| int | reserveId (int rank, int forTreeId) |
| This operation is not const as it does some internal bookkeeping. | |
| int | getRank (int treeId) const |
| You hand in a tree number and the node tells you on which rank such a tree is hosted. | |
| void | deregisterId (int id) |
| Only the SpacetreeSet should call this operation. | |
| bool | continueToRun () |
| You should call this operation only on the ranks >0 to find out whether you should do more iteration/sweeps. | |
| void | setNextProgramStep (int number) |
| The user tells the set which program step to use, i.e. | |
| int | getCurrentProgramStep () const |
| void | init () |
| Init the parallel environment. | |
| void | shutdown () |
| Shutdown the node. | |
| GridDataExchangeMetaInformation | getGridDataExchangeMetaInformation (int sendingTreeId, int receivingTreeId, ExchangeMode exchange) const |
| I use two tags per spacetree per rank: one for boundary data (horizontal) and one for up-down and synchronous (forks) data exchanges (vertical and fork/join). | |
Static Public Member Functions | |
| static void | initMPIDatatypes () |
| I originally wanted to embed these guys into the singleton's constructor. | |
| static void | shutdownMPIDatatypes () |
| static Node & | getInstance () |
| This operation returns the singleton instance. | |
| static bool | isGlobalMaster (int treeId) |
| Is this the global master? | |
| static int | getOutputStackNumberForHorizontalDataExchange (int id) |
| Hand in a spacetree id and get back the number that we should use to send something to this tree. | |
| static int | getInputStackNumberForVerticalDataExchange (int id) |
| Get the input stack where a tree writes all of its vertical data to/from when it exchanges information with id. | |
| static int | getOutputStackNumberForVerticalDataExchange (int id) |
| static int | getInputStackNumberForHorizontalDataExchange (int id) |
| Counterpart of getOutputStackNumberOfBoundaryExchange(int) | |
| static std::set< PeriodicBoundaryStackIdentifier > | getOutputStacksForPeriodicBoundaryExchange (const tarch::la::Vector< TwoPowerD, int > &flags) |
| You hand in a the flags of a vertex and you get the boundary stack identifiers including their directions. | |
| static std::string | toString (const std::set< PeriodicBoundaryStackIdentifier > &data) |
| static int | getOutputStackForPeriodicBoundaryExchange (int faceNumber) |
| Identify output stack for periodic boundary data written by face. | |
| static int | getPeriodicBoundaryExchangeInputStackNumberForOutputStack (int outputStackNumber) |
| Hand in an output stack number of a face and you get back the input stack number. | |
| static PeriodicBoundaryStackIdentifier | getPeriodicBoundaryExchangeInputStackNumberForOutputStack (PeriodicBoundaryStackIdentifier outputStackIdentifier) |
| static int | mapPeriodicBoundaryExchangeOutputStackOntoInputStack (int outputStack) |
| static bool | isHorizontalDataExchangeOutputStackNumber (int number) |
| See getOutputStackNumberOfBoundaryExchange(). | |
| static bool | isHorizontalDataExchangeInputStackNumber (int number) |
| static bool | isVerticalDataExchangeOutputStackNumber (int number) |
| static bool | isVerticalDataExchangeInputStackNumber (int number) |
| static bool | isPeriodicBoundaryExchangeOutputStackNumber (int number) |
| static bool | isStorageStackNumber (int number) |
| There are communication stacks and storage stacks. | |
| static int | getTreeNumberTiedToExchangeStackNumber (int number) |
| Gives you back the id of a communication partner, i.e. | |
| static std::string | toString (ExchangeMode mode) |
| static std::string | getSemanticsForTag (int tag) |
| I use this for debugging. | |
Static Public Attributes | |
| static constexpr int | Terminate = -2 |
| static constexpr int | MaxSpacetreesPerRank = 1024 |
| static constexpr int | UndefProgramStep = -1 |
| Value for _currentProgramStep. | |
Private Member Functions | |
| Node () | |
| The standard constructor assigns the attributes default values and checks whether the program is compiled using the -DParallel option. | |
| Node (const Node &node)=delete | |
| The copy constructor is private. | |
| int | getId (int rank, int localTreeId) const |
| Peano maps grid instance threads + mpi ranks onto global IDs through this routine. | |
| void | registerId (int id, int masterId) |
| The operation is not thread-safe as we call it only internally, i.e. | |
Static Private Member Functions | |
| static std::bitset< 2 *Dimensions > | getPeriodicBoundaryNumber (const tarch::la::Vector< TwoPowerD, int > &flags) |
| Analyse to which of the 2d faces it is adjacent. | |
Private Attributes | |
| tarch::multicore::BooleanSemaphore | _semaphore |
| MPI_Comm | _dataExchangeCommunicators [MaxSpacetreesPerRank] |
| int | _currentProgramStep |
| int | _rankOrchestrationTag |
| std::map< int, TreeEntry > | _treeEntries |
Static Private Attributes | |
| static tarch::logging::Log | _log |
| Logging device. | |
| static Node | _singleton |
| static constexpr int | StacksPerCommunicationPartner = 4 |
| We need one stack for outgoing data, one for incoming data, and we distinguish horizontal and vertical data flow. | |
Friends | |
| class | peano4::parallel::tests::NodeTest |
Node is Peano's abstraction for the hybrid of MPI/threads.
It represents one spacetree instance basically.
| typedef std::pair<int, MPI_Comm> peano4::parallel::Node::GridDataExchangeMetaInformation |
| typedef std::pair<int, int > peano4::parallel::Node::PeriodicBoundaryStackIdentifier |
A periodic boundary stack is basically a stack (an integer), but I do augment it by a bitset which identifies which symmetry axis belong to this stack number.
So if we return (23,010), then data goes to stack 23 and it is a stack that does realise periodic boundary conditions along the y axis.
The second argument (as sketched above) is of type std::bitset<Dimensions>. Unfortunately, bitsets do not work properly within the pair type as they seem not to define the comparison operator properly. So I use an int here and wrap it into a bitset whenever necessary.
|
strong |
|
private |
The standard constructor assigns the attributes default values and checks whether the program is compiled using the -DParallel option.
If this is not the case, a warning is logged.
|
privatedelete |
The copy constructor is private.
|
virtual |
The standard destructor calls MPI_Finalize().
| bool peano4::parallel::Node::continueToRun | ( | ) |
You should call this operation only on the ranks >0 to find out whether you should do more iteration/sweeps.
The spacetree set internally hijacks this operation also on rank 0 to trigger the send out of startup messages. Therefore, the operation is not const.
| void peano4::parallel::Node::deregisterId | ( | int | id | ) |
Only the SpacetreeSet should call this operation.
| int peano4::parallel::Node::getCurrentProgramStep | ( | ) | const |
| int peano4::parallel::Node::getGlobalTreeId | ( | int | treeId | ) | const |
| GridDataExchangeMetaInformation peano4::parallel::Node::getGridDataExchangeMetaInformation | ( | int | sendingTreeId, |
| int | receivingTreeId, | ||
| ExchangeMode | exchange ) const |
I use two tags per spacetree per rank: one for boundary data (horizontal) and one for up-down and synchronous (forks) data exchanges (vertical and fork/join).
The nice thing about Peano's new data management is that it is stack-only. Furthermore, all stack data exchange is triggered via the spacetree set, i.e. the order of the stacks is known externally, too. So we effectively do not need that many tags even though we need different tags per tree pair.
|
private |
Peano maps grid instance threads + mpi ranks onto global IDs through this routine.
To identify a global stack number, please use this function and add it to your local stack number.
|
static |
Counterpart of getOutputStackNumberOfBoundaryExchange(int)
|
static |
Get the input stack where a tree writes all of its vertical data to/from when it exchanges information with id.
Such vertical data is multiscale information or split/join data.
|
static |
This operation returns the singleton instance.
Before using this instance, one has to call the init() operation on the instance returned.
Referenced by peano4::maps::HierarchicalStackMap< T >::clear(), peano4::maps::STDStackMap< T >::clear(), peano4::maps::HierarchicalStackMap< T >::empty(), peano4::maps::HierarchicalStackMap< T >::garbageCollection(), peano4::maps::STDStackMap< T >::garbageCollection(), peano4::maps::HierarchicalStackMap< T >::getForPop(), peano4::maps::HierarchicalStackMap< T >::getForPush(), peano4::maps::HierarchicalStackMap< T >::getKeys(), and peano4::maps::HierarchicalStackMap< T >::holdsStack().
| int peano4::parallel::Node::getLocalTreeId | ( | int | treeId | ) | const |
Referenced by peano4::maps::HierarchicalStackMap< T >::clear(), peano4::maps::STDStackMap< T >::clear(), peano4::maps::HierarchicalStackMap< T >::empty(), peano4::maps::HierarchicalStackMap< T >::garbageCollection(), peano4::maps::STDStackMap< T >::garbageCollection(), peano4::maps::HierarchicalStackMap< T >::getForPop(), peano4::maps::HierarchicalStackMap< T >::getForPush(), and peano4::maps::HierarchicalStackMap< T >::holdsStack().
| int peano4::parallel::Node::getNumberOfRegisteredTrees | ( | ) | const |
|
static |
Identify output stack for periodic boundary data written by face.
To make the mapping of face and vertex stacks consistent, we take the face number and determine what the adjacent, i.e. face-connected neighbour volume out of the \( 3^d \) adjacent, i.e. vertex-connected volumes is.
Take care that the result is not a mere linearisation of the index. You have to explicitly exclude the centre volume.
|
static |
Hand in a spacetree id and get back the number that we should use to send something to this tree.
|
static |
|
static |
You hand in a the flags of a vertex and you get the boundary stack identifiers including their directions.
We first analyse which faces with a periodic boundary flag are adjacent to the vertex. For this, we employ getPeriodicBoundaryNumber(). After that, we build the power set over the result: If a face is adjacent to the upper face and the left face, it also has to communicate along the diagonal, i.e. the logical combination of the two.
We now have to translate these powerset entries into stack indices. For this, we run over the \( 3^d \) neighbour cells or a cell, construct the index for this direction, and then look whether this index is contained within the power set.
For the powerset, I'd like to use
std::set< std::bitset<2*Dimensions> > powerSet;
but C++ defines no comparison operator on bitsets and therefore you can't use it in a set.
|
static |
Hand in an output stack number of a face and you get back the input stack number.
If you want to use it for merging data, you have to be careful: Think about the right face of your domain. It is assigned a BC output stack. This routine tells you onto which input stack this output stack is mapped. In return, if you read in the periodic BC data for the right face, you will merge a data with its own! You however have to merge it with the input BC stack from the left face.
This routine is not made to orchestrate the stack exchange! It is meant that you hand in number to which a vertex or face would write to. And then you get the number back which it should read from. This is not how the stacks are permuted after each step! If you want to achieve this, you have to call mapPeriodicBoundaryExchangeOutputStackOntoInputStack().
|
static |
|
staticprivate |
Analyse to which of the 2d faces it is adjacent.
We enumerate again in the way we always enumerate, i.e. first the face with the x axis as normal, then the one with the y axis, ... And after these d faces, we enumerate all the opposite faces on the cube.
The routine is non-trivial: We store the \( 2^d \) adjacency flags per vertex, and now have to work backwards which periodic faces they do represent.
| int peano4::parallel::Node::getRank | ( | int | treeId | ) | const |
You hand in a tree number and the node tells you on which rank such a tree is hosted.
|
static |
I use this for debugging.
When I have a tag, I can ask the node whether the node is aware what this tag actually means.
|
static |
Gives you back the id of a communication partner, i.e.
a spacetree number, which corresponds to stack number. This works for all kinds of stacks.
| void peano4::parallel::Node::init | ( | ) |
Init the parallel environment.
Every rank has to call this routine. In Peano 4, the routine peano4::initSingletons() is responsible for this. If you don't use it, ensure that each and every MPI rank calls this routine relatively early throughout the execution. An example for the explicit usage can be found in src/unittest/main.cpp.
The counterpart is shutdown().
|
static |
I originally wanted to embed these guys into the singleton's constructor.
However, the singleton might not yet be up when I run the (mpi) tests.
|
static |
Is this the global master?
|
static |
|
static |
See getOutputStackNumberOfBoundaryExchange().
|
static |
|
static |
There are communication stacks and storage stacks.
This is a storage stack.
Referenced by peano4::maps::HierarchicalStackMap< T >::garbageCollection(), and peano4::maps::STDStackMap< T >::garbageCollection().
|
static |
|
static |
|
static |
|
private |
The operation is not thread-safe as we call it only internally, i.e.
you are not supposed to call this function directly.
| int peano4::parallel::Node::reserveId | ( | int | rank, |
| int | forTreeId ) |
This operation is not const as it does some internal bookkeeping.
It internally invokes registerId() on the result.
| forTreeId | If we run a split, this is the rank that wants to split. In general, this always is the parent/master of the new rank. So if you move a rank, you have to ensure that the master is preserved. |
| void peano4::parallel::Node::setNextProgramStep | ( | int | number | ) |
The user tells the set which program step to use, i.e.
which adapter to use for the next grid run or which step to run next. Should only be called on global master.
| void peano4::parallel::Node::shutdown | ( | ) |
Shutdown the node.
This is the counterpart of The shutdown is invoked by peano4::shutdownSingletons()
If we run it on the global master, we set the program state to terminate and distribute this message. After that, we wait in a barrier. If we are
For this, we hijack the routine continueToRun(). This is really a hack. We in particular have to be careful and may not call the continue operation on the other ranks. They typically are stuck in a continue while loop in their main routine and just wait for the global master to send out this terminate command.
It is important that this is the very first call when you want your application to go down. Invoked on the global master, it triggers the shutdown on all the other ranks. You will get deadlocks if you try to free any common memory (RDMA) or free datatypes if you don't call the node's shutdown first.
Other ranks might struggle to receive the shutdown command as they might still ask for further subtrees, e.g., or issue load balancing. Therefore, I insert a Peano 4 barrier right after we've issued the shutdown. It makes the rank that goes down handle all incoming data until we can be sure that everybody is in shutdown mode.
Before you use this routine, you have to ensure that the node is properly initialised:
An example for this can be found in src/unittests/main.cpp. In Peano 4, you usually don't care about the initialisation as the function peano4::initSingletons() will handle it. Compare to the documentation of init().
The shutdown process is slightly more complicated than a simple function call. The global master (rank 0) has to call shutdown(). At this point, we assume that the other ranks are in the continue loop:
Rank 0 (global master) will send out the terminate message and then all the other ranks will receive this message and hop out of the continueToRun() loop. After that, they have to call shutdown() as well. Eventually, each rank will be in the barrier within shutdown(), so we'll all leave shutdown() and can continue (to shut down the system).
|
static |
|
static |
|
static |
|
friend |
|
private |
|
staticprivate |
|
private |
|
private |
|
staticconstexpr |
|
staticconstexprprivate |
|
staticconstexpr |
|
staticconstexpr |
1.10.0