Passing a pointer to MPI_Win_allocate_shared through a wrapper

Some reading of the MPI standard and understanding of C++ pointers and function arguments may help here. For example, reading the Open MPI manual page for MPI_Win_allocate_shared:

On each process, it allocates memory of at least size bytes that is shared among all processes in comm, and returns a pointer to the locally allocated segment in baseptr that can be used for load/store accesses on the calling process.

The only way for a C++ function to return a value through a function argument is if that argument is a reference or a pointer to a location for the value. Therefore, although the manual page lists baseptr as void *, it is really of type void **.

Now, the difference between:

void foo(void *bar) {
   MPI_Win_allocate_shared(..., &bar, ...);
}

void *baz;
foo(baz);

and

void foo(void **bar) {
   MPI_Win_allocate_shared(..., bar, ...);
}

void *baz;
foo(&baz);

Although in both cases the call to MPI_Win_allocate_shared ends up having a void ** as an argument, the former case is conceptually wrong. You are not passing the address of baz, but rather a pointer to the formal argument bar, which holds a copy of the value of baz. The semantics of formal arguments is basically that of local variables initialised with the actual function argument:

void *bar = baz;
MPI_Win_allocate_shared(..., &bar, ...);

This writes a new value into bar while leaving the value of baz unchanged. Which is why you see a new value after the call to MPI_Win_allocate_shared inside MPI_WRAP but the old value once it returns to the calling function.

The latter is akin to

void **bar = &baz;
MPI_Win_allocate_shared(..., bar, ...);

This has totally different semantics. bar now contains the address of baz and that’s the location MPI_Win_allocate_shared writes the address of the allocated buffer into.

So the correct C/C++ code is:

void foo(void **bar) {
   MPI_Win_allocate_shared(..., bar, ...);
}

void *baz;
foo(&baz);

C++ has references and the same can be written like so:

void foo(void *&bar) {
   MPI_Win_allocate_shared(..., &bar, ...);
}

void *baz;
foo(baz);

This is very similar to the first (incorrect) case, with the crucial difference that here the formal argument bar is an alias of the actual argument baz, so now &bar is the same as &baz. Hence, MPI_Win_allocate_shared will write the return value into the storage space of baz.

A side note: I recommend the manual pages from Open MPI because the explanations there are basically excerpts from the MPI standard.