Defect: MPI_Type_extent, type of 2nd argument is pointer to MPI_Aint

[jbmaggard]

Defect/Bug Report

Defect: MPI_Type_extent, type of 2nd argument is pointer to MPI_Aint

• OpenCoarrays Version: 1.9.1
• Fortran Compiler: GNU Fortran (x86_64-posix-seh-rev2, Built by MinGW-W64 project) 7.1.0
• C compiler used for building lib: gcc (x86_64-posix-seh-rev2, Built by MinGW-W64 project) 7.1.0
• Installation method: Manual, Win7-64
• Output of ver: Microsoft Windows [Version 6.1.7601]
• MPI library being used: Intel(R) MPI Library for Windows* OS, Version 2017 Update 3 Build 20170405
• Machine architecture and number of physical cores: Intel Haswell i7, 4 cores, 8 threads
• Version of CMake: N/A

Observed Behavior

C:\TEMP\oca\src_1.9.1\mpi>gcc --version
gcc (x86_64-posix-seh-rev2, Built by MinGW-W64 project) 7.1.0

• -I.. used to find libcaf.h
• -I../.. used to find mpi.h
• -D_POSIX used to find SIGKILL in signal.h
• -DALLOCA_MISSING because it is missing
• -DUSE_GCC for correct typedef of MPI_Aint in mpi.h (long long int)
• -g -Og for debugging with gdb
• -m64 to give an error message if compile attempted with Win32 version of mingw-w64 gcc

C:\TEMP\oca\src_1.9.1\mpi>gcc -g -m64 -DGCC_GE_7 -DPREFIX_NAME=_gfortran_caf_ -DUSE_FAILED_IMAGES -DALLOCA_MISSING -D_POSIX -DUSE_GCC -Og -c mpi_caf.c -I .. -I ../..
mpi_caf.c: In function 'redux_char_by_reference_adapter':
mpi_caf.c:4140:30: warning: passing argument 2 of 'MPI_Type_extent' from incompatible pointer type [-Wincompatible-pointer-types]
   MPI_Type_extent(*datatype, &string_len);
                              ^
In file included from mpi_caf.c:44:0:
../../mpi.h:1121:5: note: expected 'MPI_Aint * {aka long long int *}' but argument is of type 'long int *'
 int MPI_Type_extent(MPI_Datatype datatype, MPI_Aint *extent);
     ^~~~~~~~~~~~~~~

In linux (which is LP64), the correct typedef of MPI_Aint is long int (see mpi.h from MPICH 3.2). For Windows (which is LLP64), the mpi.h from the Intel MPI Windows SDK defines -DUSE_GCC to implement the correct typedef of long long int for MPI_Aint with GCC.

Intel MPI Windows SDK, mpi.h:  int MPI_Type_extent(MPI_Datatype datatype, MPI_Aint *extent);

C:\TEMP\oca\src_1.9.1\mpi>fc /L /N mpi_caf.c mpi_caf.c.original
Comparing files mpi_caf.c and MPI_CAF.C.ORIGINAL
***** mpi_caf.c
 4142:  {
 4143:    MPI_Aint string_len;
 4144:    MPI_Type_extent(*datatype, &string_len);
***** MPI_CAF.C.ORIGINAL
 4142:  {
 4143:    long int string_len;
 4144:    MPI_Type_extent(*datatype, &string_len);
*****

Question

I posted details of my build (including build of import library for impi.dll) and results at people.tamu.edu/~bmaggard, in case anyone is interested in a native Win64 CAF solution (Win7-64; mingw-w64 gfortran 7.1.0.rev2; Intel MPI Runtime 2017 Update 3; OpenCoarrays-1.9.1). Execution results are equivalent to building 1.9.1 on linux with install.sh (Fedora 26; gcc-gfortran 7.1.1-3, MPICH 3.2, OpenCoarrays-1.9.1).

For GCC 7.1.0, my question at this time is whether I should be using any/all of the defines: COMPILER_SUPPORTS_CAF_INTRINSICS, STRIDED, and COMPILER_SUPPORTS_ATOMICS. Feedback appreciated on this question.

[zbeekman]

N.B.: Officially, we do not support Windows, because we don't have good access to windows machines, and are short on manpower and funding. However, we welcome any contributions to improve our unofficial windows support.

I can't remember the details, but I seem to recall that MS MPI may have some issues with MPI-3 RMA. Try searching through our closed issues for "windows" to see if you can find any relevant clues. Also, perhaps @jeffhammond or @afanfa knows more about this.

As far as appropriate defines go, for GCC 7.2 (would be same for 7.1) on mac OS, CMake generates the following compilation lines:

opencoarrays.F90: (NOT needed for GCC 7.1, unless you wish to use some special extensions)

/usr/local/bin/gfortran-7 -DGCC_GE_7 -DMPI_WORKING_MODULE -DPREFIX_NAME=_gfortran_caf_ -DUSE_FAILED_IMAGES -I/Users/ibeekman/Sandbox/opencoarrays-clean/src -I/usr/local/Cellar/mpich/3.2_3/include -I/Users/ibeekman/Sandbox/opencoarrays-clean/build/mod  -O3 -DNDEBUG -O3 -J../../mod   -c /Users/ibeekman/Sandbox/opencoarrays-clean/src/extensions/opencoarrays.F90 -o CMakeFiles/caf_mpi.dir/__/extensions/opencoarrays.F90.o

mpi_caf.c: (should be all you really need...)

/usr/local/bin/gcc-7 -DGCC_GE_7 -DMPI_WORKING_MODULE -DPREFIX_NAME=_gfortran_caf_ -DUSE_FAILED_IMAGES -I/Users/ibeekman/Sandbox/opencoarrays-clean/src -I/usr/local/Cellar/mpich/3.2_3/include -I/Users/ibeekman/Sandbox/opencoarrays-clean/build/mod  -O3 -DNDEBUG   -o CMakeFiles/caf_mpi.dir/mpi_caf.c.o   -c /Users/ibeekman/Sandbox/opencoarrays-clean/src/mpi/mpi_caf.c

For GCC 7.1.0, my question at this time is whether I should be using any/all of the defines: COMPILER_SUPPORTS_CAF_INTRINSICS, STRIDED, and COMPILER_SUPPORTS_ATOMICS. Feedback appreciated on this question.

Here is a guide to the defines you should and should not be using:

• GCC_GE_7 Set this define if building against GFortran >= 7.0
• MPI_WORKING_MODULE Unless the windows MPI SDK ships with an mpi.mod generated by GFortran 7.x you should NOT set the define. (It shouldn't have any impact on mpi_caf.c, however.)
• PREFIX_NAME should be set to _gfortran_caf_ when compiling for GFortran >= 5.2, should be the same on Windows
• USE_FAILED_IMAGES should NOT be set, unless you're confident that windows MPI has support for experimental/proposed MPI features required to support fault tolerant failed images in Fortran 2015(?). I would strongly recommend NOT defining this for your windows build. Right now, there is a private fork of the OpenMPI fault tolerance project that has working experimental features to support failed images and MPICH 3.2. I am unaware of any other MPI implementations with sufficient support for the proposed experimental MPI features required to be able to enable failed images in OpenCoarrays.
• STRIDED is for an experimental more efficient transfer of strided coarrays. I'm not sure if the implementation is 100% complete, @afanfa probably knows.
• COMPILER_SUPPORTS_CAF_INTRINSICS is not needed except for old GCC/GFortran <= 4.9
• COMPILER_SUPPORTS_ATOMICS seems to be outdated (it looks like we need to update the documentation in INSTALL.md too) to add experimental events support in GCC/GFortran <= 6 or 7, which now has native events support, I'm pretty sure. (Events were partially implemented via atomics in the coarrayfortran.F90 extensions module.) This should definitely NOT be set for GFortran 7.1.

@jbmaggard Is my understanding correct that by applying the patch wherein long int string_len; ==> MPI_Aint string_len; and adding -D_POSIX and -DUSE_GCC allow you to successfully build and run OpenCoarrays 1.9.1 on windows against Intel(R) MPI Library for Windows* OS, Version 2017 Update 3 Build 20170405? ❗️ 🎉

[zbeekman]

Also, FYI, there is a script to build OpenCoarrays using WSL at https://github.com/sourceryinstitute/OpenCoarrays/blob/master/windows-install.sh

[zbeekman]

Also, this def looks like a bug in OpenCoarrays: https://www.mpich.org/static/docs/v3.2/www3/MPI_Type_extent.html

[zbeekman]

@jbmaggard one additional question: Do you anticipate those defines need to always be set on Windows? (i.e. for cygwin or other non mingw windows environments? Or other MPI implementations?) Could they ever cause any harm if set when Windows is detected?

[zbeekman]

@jbmaggard please let me know if https://github.com/sourceryinstitute/OpenCoarrays/pull/436/files looks like it will fix the issue.

[jbmaggard]

@zbeekman Affirmative. Using the Intel MPI Runtime, along with an import library to impi.dll allows build and run of all the src/tests examples (Win64 native, not cygwin or WSL). Results are comparable to what I get on linux by cafrun'ing (all of) the executables built by install.sh.

@zbeekman Thanks for the comments on compiler defines. I started by doing a cmake, make clean, make VERBOSE=1 on linux to see exactly what gfortran, gcc, and ar command lines were being used for compile and link.

-D_POSIX is a mingw-w64 issue. You will see the #ifdef if you take a look at signal.h. Specifically, it was needed to have #include <signal.h> in mpi_caf.c define SIGKILL.

From intel mpi.h, on -DUSE_GCC to typedef MPI_Aint as long long int:
#ifdef MPI_AINT64_TYPE
#undef MPI_AINT64_TYPE
#endif
#if defined(USE_GCC) || defined(GNUC)
#define MPI_AINT64_TYPE long long
#else
#define MPI_AINT64_TYPE __int64
#endif
typedef MPI_AINT64_TYPE MPI_Aint;
#undef MPI_AINT64_TYPE

From mpi_caf.c on alloca.h; functionality is apparently provided elsewhere by other includes for mingw-w64:
#ifndef ALLOCA_MISSING
#include <alloca.h> /* Assume functionality provided elsewhere if missing */
#endif

[jbmaggard]

@zbeekman On the src/mpi/CMakeLists.txt changes you propose, I think it may be a little more complicated than that.

1. My success was with mingw-w64 GCC, not TDM or mingw (specifically about -D_POSIX)
2. Defining MPI_Aint in Intel MPI's mpi.h compatible with mingw-w64 gcc is why I used -DUSE_GCC
3. You have to build an import library to the Intel MPI runtime's impi.dll (I called it libimpi.a) using the gendef and dlltool utilities of mingw-w64.
4. The mingw-w64 /bin and Intel MPI /bin both have to be in the path of the cmd window, for the gfortran runtime and the Intel MPI runtime.

[jeffhammond]

@zbeekman I do not consider Windows to be a relevant platform for parallel computing and know very little about how various parallel programming tools behave in that environment. My standard recommendation for using HPC tools on Windows is to install a Debian VM in Virtual Box. My attempts to use WSL in Windows 10 have proven unsuccessful and frustrating.

[zbeekman]

@jeffhammond sure, I guess I miss-remembered you being part of a past discussion. Thanks for the input!

@jbmaggard What do you suggest I do RE: PR #436? The change to mpi_caf.c appears to be required no matter what, and a bonafide bug, judging from various MPI documentation. I don't entirely follow your comments above, since I have zero relevant experience with Windows, and I don't know or understand all the differences between mingw, mingw-w64, TDM (?)

Am I understanding you correctly that:

1. -D_POSIX is mingw(-w64?) specific? CMake provides functionality to test for mingw as listed in the Variables that describe the system section of their documentation. Is it a reasonable guess that we should define _POSIX for any mingw variant?
2. -DUSE_GCC is needed by Intel MPI? So we should really be detecting Intel MPI rather than windows to decide whether we need to add this definition? Do you know if this is true on *nix platforms too? If so any ideas on how to do system introspection to detect the Intel MPI runtime? (@jeffhammond do you have any thoughts on this? Feel free to ignore my inquiries if you're not inclinde to, BTW)

As far as alloca.h goes it's for allocating memory on the stack thats automatically freed, and different platforms provide this functionality in different ways. For instance FreeBSD includes this in libc: https://www.freebsd.org/cgi/man.cgi?query=alloca&apropos=0&sektion=0&manpath=FreeBSD%206.1-RELEASE&format=html