Peano 4
No Matches
tarch::mpi::BooleanSemaphore::BooleanSemaphoreService Class Reference

#include <BooleanSemaphore.h>

Inheritance diagram for tarch::mpi::BooleanSemaphore::BooleanSemaphoreService:
Collaboration diagram for tarch::mpi::BooleanSemaphore::BooleanSemaphoreService:

Data Structures

struct  SemaphoreMapEntry

Public Member Functions

void init ()
virtual void shutdown () override
 ~BooleanSemaphoreService ()=default
 Destructor of the service.
virtual void receiveDanglingMessages () override
 Receive any lock-related dangling messages.
void acquireLock (int number)
 Acquire the lock.
void releaseLock (int number)
 Release a lock.
int getNumberOfLockedSemaphores ()
std::string toString () const
- Public Member Functions inherited from tarch::services::Service
virtual ~Service ()

Static Public Member Functions

static BooleanSemaphoreServicegetInstance ()
 Don't use this routine.

Private Member Functions

 BooleanSemaphoreService ()=default
 BooleanSemaphoreService (const BooleanSemaphoreService &)=delete
void serveLockRequests ()
 Serve pending lock requests.
bool tryLockSemaphoreOnGlobalMaster (int number, int forRank)
 Try to lock a semaphore.
void lockSemaphoreOnGlobalMaster (int number, int forRank)
void unlockSemaphoreOnGlobalMaster (int number, int forRank)
 Unlock semaphore on global master.
void addMapEntryLazily (int number)
 Add map entry.

Private Attributes

int _semaphoreTag
 Tag used to exchange locking MPI messages.
tarch::multicore::BooleanSemaphore _mapAccessSemaphore
 Semaphore for the global master's map of locks.
tarch::multicore::BooleanSemaphore _reserverationRequestsSemaphore
std::map< int, SemaphoreMapEntry_map
 Map of semaphores.
std::vector< std::pair< int, int > > _pendingLockRequests
 List of pending lock requests.

Static Private Attributes

static BooleanSemaphoreService _singleton

Additional Inherited Members

- Protected Attributes inherited from tarch::services::Service
tarch::multicore::RecursiveSemaphore _receiveDanglingMessagesSemaphore
 Recursive semaphores.

Detailed Description

Definition at line 73 of file BooleanSemaphore.h.

Constructor & Destructor Documentation

◆ BooleanSemaphoreService() [1/2]

tarch::mpi::BooleanSemaphore::BooleanSemaphoreService::BooleanSemaphoreService ( )

◆ BooleanSemaphoreService() [2/2]

tarch::mpi::BooleanSemaphore::BooleanSemaphoreService::BooleanSemaphoreService ( const BooleanSemaphoreService & )

◆ ~BooleanSemaphoreService()

tarch::mpi::BooleanSemaphore::BooleanSemaphoreService::~BooleanSemaphoreService ( )

Destructor of the service.

As the service is a singleton and a service (hence the name), it has to deregister itself. Otherwise, the overall service landscape still might try to call receiveDanglingMessages().

Member Function Documentation

◆ acquireLock()

void tarch::mpi::BooleanSemaphore::BooleanSemaphoreService::acquireLock ( int number)

◆ addMapEntryLazily()

void tarch::mpi::BooleanSemaphore::BooleanSemaphoreService::addMapEntryLazily ( int number)

Add map entry.

Only defined/used on global master. We do not maintain a map of all existing semaphores, but build up a dictionary of these guys at runtime in a lazy fashion.

This routine is not thread-safe, i.e. I assume that somebody calling this routine has ensured that no data races occur and notably that noone uses _map meanwhile.

numberEvery boolean semaphore has a globally unique positive number. The MPI messages we send around carry this number if we want to acquire a lock. They carry the negative number if we want to release a lock. To allow for these sign flips, number may not equal zero.

Definition at line 206 of file BooleanSemaphore.cpp.

References _map, and assertion1.

Referenced by tryLockSemaphoreOnGlobalMaster().

Here is the caller graph for this function:

◆ getInstance()

tarch::mpi::BooleanSemaphore::BooleanSemaphoreService & tarch::mpi::BooleanSemaphore::BooleanSemaphoreService::getInstance ( )

Don't use this routine.

It returns the global semaphore and is only used on the master/by the core.

Definition at line 201 of file BooleanSemaphore.cpp.

References _singleton.

Referenced by tarch::mpi::BooleanSemaphore::enterCriticalSection(), peano4::initSingletons(), and peano4::shutdownSingletons().

Here is the caller graph for this function:

◆ getNumberOfLockedSemaphores()

int tarch::mpi::BooleanSemaphore::BooleanSemaphoreService::getNumberOfLockedSemaphores ( )

Definition at line 191 of file BooleanSemaphore.cpp.

References _pendingLockRequests, _reserverationRequestsSemaphore, and tarch::mpi::Rank::getInstance().

Here is the call graph for this function:

◆ init()

void tarch::mpi::BooleanSemaphore::BooleanSemaphoreService::init ( )

Definition at line 50 of file BooleanSemaphore.cpp.

References _semaphoreTag, tarch::services::ServiceRepository::addService(), tarch::services::ServiceRepository::getInstance(), and tarch::mpi::Rank::reserveFreeTag().

Referenced by peano4::initSingletons().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ lockSemaphoreOnGlobalMaster()

void tarch::mpi::BooleanSemaphore::BooleanSemaphoreService::lockSemaphoreOnGlobalMaster ( int number,
int forRank )

Global master and deadlocks

We always have to call receiveDanglingMessages() quite aggressively. On the one hand, we might try to lock a semaphore which is already locked by another rank. We have to give that other rank the opportunity to free the lock again. If we don't allow this to happen, we will never get the opportunity to lock this semaphore for our own rank, i.e. the global master. s * There is another reason for this polling, too: It is important that we don't prematurely lock the semaphore if we are called on the master. We should always try to serve other ranks first to avoid massive MPI rank divergence. Further to that, we take the flag _localRankLockRequestSemaphore into account. If it is set, we know that each rank will request the semaphore at least once. Consequently, we do acquire it if and only if another rank has grabbed it before. Basically, we try to prioritise all other ranks and let locks by rank 0 pass through last. Please consult the class documentation on potential deadlocks.

Definition at line 95 of file BooleanSemaphore.cpp.

References _map, assertion, tarch::mpi::Rank::getInstance(), logDebug, logWarning, receiveDanglingMessages(), tarch::mpi::Rank::setDeadlockTimeOutTimeStamp(), tarch::mpi::Rank::setDeadlockWarningTimeStamp(), and tryLockSemaphoreOnGlobalMaster().

Referenced by acquireLock().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ receiveDanglingMessages()

void tarch::mpi::BooleanSemaphore::BooleanSemaphoreService::receiveDanglingMessages ( )

Receive any lock-related dangling messages.

This routine polls and if there's a release or request message, it tries to answer it.

Usage within multithreaded environment

We rely on the fact that no multiple threads can access the receiveDangling thing at the same time if you go through the services. In return, please do never invoke this routine directly: If two threads jump into receiveDangingMessages, they might issue the Iprobe one after another and get a "yes" - even though there is only one message in the queue. Consequently, both threads bump into the receive operation. Now, only one message is there and, consequently, the second thread deadlocks.


The message polls for messages. Messages with a positive value mean the system tries to acquire an MPI lock. Messages with a negative value mean that the corresponding ranks wants to release the lock with the corresponding positive value.

Release requests are handled immediately by calling releaseLock(). Lock requests are not handled direclty. Instead, we enqueue them into _pendingLockRequests. Eventually, I serve them by invoking serveLockRequests(). This is a function I call if and only if no further messages are pending. Therefore, this code fragments prioritises frees over locks.

Implements tarch::services::Service.

Definition at line 159 of file BooleanSemaphore.cpp.

References _pendingLockRequests, _reserverationRequestsSemaphore, _semaphoreTag, assertion, tarch::mpi::Rank::getCommunicator(), tarch::mpi::Rank::getInstance(), logDebug, releaseLock(), serveLockRequests(), and toString().

Referenced by lockSemaphoreOnGlobalMaster().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ releaseLock()

void tarch::mpi::BooleanSemaphore::BooleanSemaphoreService::releaseLock ( int number)

Release a lock.

This routine delegates the call to unlockSemaphoreOnGlobalMaster() if we run it on the global master. To avoid rank divergence, we immediately serve follow-up lock requests after we've freed the lock.

If we are not on the global master, we send an unlock message to this master and return. There will be no confirmation or similar.

Definition at line 246 of file BooleanSemaphore.cpp.

References _semaphoreTag, assertion, assertionMsg, tarch::mpi::Rank::getGlobalMasterRank(), tarch::mpi::Rank::getInstance(), logDebug, serveLockRequests(), and unlockSemaphoreOnGlobalMaster().

Referenced by receiveDanglingMessages().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ serveLockRequests()

void tarch::mpi::BooleanSemaphore::BooleanSemaphoreService::serveLockRequests ( )

Serve pending lock requests.

If a remote rank does request a lock, I don't reserve it straightway. Instead, I buffer it in _pendingLockRequests. This buffering is realised within receiveDanglingMessages().

From time to time, I now have to check whether I can serve the request. I do so in receiveDanglingMessages() whenever there are no more messages to be appended to the list of locks, and I also try to serve requests directly after someone has released a lock, hoping that this is the perfect timing.

The routine serves delegates the actual decision whether to lock or not to tryLockSemaphoreOnGlobalMaster(), but it can serve multiple lock requests. This makes sense, as the global master manages all sempahores and not only one, i.e. it might be able to serve multiple requests. Actually, to avoid deadlock scenarios, it is furthermore important that the routine runs through all requests.

Definition at line 132 of file BooleanSemaphore.cpp.

References _pendingLockRequests, _reserverationRequestsSemaphore, _semaphoreTag, assertion, tarch::mpi::Rank::getCommunicator(), tarch::mpi::Rank::getInstance(), logDebug, toString(), and tryLockSemaphoreOnGlobalMaster().

Referenced by receiveDanglingMessages(), and releaseLock().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ shutdown()

void tarch::mpi::BooleanSemaphore::BooleanSemaphoreService::shutdown ( )

Implements tarch::services::Service.

Definition at line 55 of file BooleanSemaphore.cpp.

References tarch::services::ServiceRepository::getInstance(), and tarch::services::ServiceRepository::removeService().

Referenced by peano4::shutdownSingletons().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ toString()

std::string tarch::mpi::BooleanSemaphore::BooleanSemaphoreService::toString ( ) const

Definition at line 59 of file BooleanSemaphore.cpp.

References _map, and _pendingLockRequests.

Referenced by receiveDanglingMessages(), serveLockRequests(), and unlockSemaphoreOnGlobalMaster().

Here is the caller graph for this function:

◆ tryLockSemaphoreOnGlobalMaster()

bool tarch::mpi::BooleanSemaphore::BooleanSemaphoreService::tryLockSemaphoreOnGlobalMaster ( int number,
int forRank )

Try to lock a semaphore.

This routine is only called on the global master. We try to lock the global semaphore number for forRank. If that works, we make a note for which rank we have locked it. Before we however check if we can serve a request, we invoke addMapEntryLazily(), i.e. we ensure that there is an entry in the database for forRank.

Lock request has been served successfully

Definition at line 75 of file BooleanSemaphore.cpp.

References _map, _mapAccessSemaphore, addMapEntryLazily(), assertion, tarch::mpi::Rank::getInstance(), and logDebug.

Referenced by lockSemaphoreOnGlobalMaster(), and serveLockRequests().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ unlockSemaphoreOnGlobalMaster()

void tarch::mpi::BooleanSemaphore::BooleanSemaphoreService::unlockSemaphoreOnGlobalMaster ( int number,
int forRank )

Unlock semaphore on global master.

Very simple implementation:

  • Acquire a lock on the global semaphore map
  • Check that all data is consistent, i.e. someone has actually requested this lock before
  • Free the lock, i.e. set the flag to "not locked"

Definition at line 118 of file BooleanSemaphore.cpp.

References _map, _mapAccessSemaphore, assertion, assertion2, assertionEquals, tarch::mpi::Rank::getInstance(), logDebug, and toString().

Referenced by releaseLock().

Here is the call graph for this function:
Here is the caller graph for this function:

Field Documentation

◆ _map

std::map<int,SemaphoreMapEntry> tarch::mpi::BooleanSemaphore::BooleanSemaphoreService::_map

Map of semaphores.

This is the actual map which stores per semaphore number a bool that says whether it is currently taken. So false means noone's requested it, true means someone holds it.

Definition at line 124 of file BooleanSemaphore.h.

Referenced by addMapEntryLazily(), lockSemaphoreOnGlobalMaster(), toString(), tryLockSemaphoreOnGlobalMaster(), and unlockSemaphoreOnGlobalMaster().

◆ _mapAccessSemaphore

tarch::multicore::BooleanSemaphore tarch::mpi::BooleanSemaphore::BooleanSemaphoreService::_mapAccessSemaphore

Semaphore for the global master's map of locks.

On the master, I have to be sure that no two threads do access the map of semaphores concurrently. Therefore, I need a shared memory semaphore.

Definition at line 89 of file BooleanSemaphore.h.

Referenced by tryLockSemaphoreOnGlobalMaster(), and unlockSemaphoreOnGlobalMaster().

◆ _pendingLockRequests

std::vector< std::pair<int, int> > tarch::mpi::BooleanSemaphore::BooleanSemaphoreService::_pendingLockRequests

List of pending lock requests.

If another rank requests a semaphore, I store this request temporarily here. Each entry is a map from a rank that requests access to the number of the semaphore it is asking for.

Definition at line 134 of file BooleanSemaphore.h.

Referenced by getNumberOfLockedSemaphores(), receiveDanglingMessages(), serveLockRequests(), and toString().

◆ _reserverationRequestsSemaphore

tarch::multicore::BooleanSemaphore tarch::mpi::BooleanSemaphore::BooleanSemaphoreService::_reserverationRequestsSemaphore

◆ _semaphoreTag

int tarch::mpi::BooleanSemaphore::BooleanSemaphoreService::_semaphoreTag

Tag used to exchange locking MPI messages.

Definition at line 80 of file BooleanSemaphore.h.

Referenced by acquireLock(), init(), receiveDanglingMessages(), releaseLock(), and serveLockRequests().

◆ _singleton

tarch::mpi::BooleanSemaphore::BooleanSemaphoreService tarch::mpi::BooleanSemaphore::BooleanSemaphoreService::_singleton

Definition at line 75 of file BooleanSemaphore.h.

Referenced by getInstance().

The documentation for this class was generated from the following files: